Back to search

University of British Columbia

On the efficiency and privacy of foundation models

Abstract

dc:description

The task of generative modeling – building models which directly capture and represent distributions, has seen a surge of popularity in recent years. The massive-scale foundation models which underpin the advances in the field have incredible performance, but come at the cost of large computational resource requirements and low privacy. The attention mechanism used in large language models requires quadratic time with respect to the length of the input sequence, in contrast with the remainder of the network requiring linear time. We propose a Simhash-based attention scheme which approximates full attention while providing tuneable parameters which trades off runtime for accuracy. Preliminary results suggest that O(n log n) time is sufficient for a good approximation of full attention. Furthermore, our method naturally admits an implementation using O(n) memory, matching the memory cost of other space-efficient attention mechanisms. It has been shown that large-scale diffusion models can be better at memorizing its training data than GANs, the previous state-of-the-art in image generative modeling. This poses significant privacy risks. We combine previous advances in differentially private machine learning and apply it to latent diffusion models (LDMs). Specifically, we use DP-SGD to train a subset of parameters in the attention modules of the model. We make comparisons with existing works using Fréchet inception distance (FID), a perceptual distance metric, and validation-set classification accuracies of downstream classifier models trained on synthetic data.

Degree

thesis:*
Name thesis:degree_name
Master of Science - MSc
Level thesis:degree_level
master's
Discipline thesis:degree_discipline
Computer Science
Grantor dc:publisher
University of British Columbia
Year dc:date
2024

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Liu, Michael F.
Contributors dc:contributor
  • Harvey, Nicholas J. A.

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/89134

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Liu, Michael F.. On the efficiency and privacy of foundation models. master's thesis, University of British Columbia, 2024. http://hdl.handle.net/2429/89134