Abstract
dc:descriptionThe task of generative modeling – building models which directly capture and represent distributions, has seen a surge of popularity in recent years. The massive-scale foundation models which underpin the advances in the field have incredible performance, but come at the cost of large computational resource requirements and low privacy. The attention mechanism used in large language models requires quadratic time with respect to the length of the input sequence, in contrast with the remainder of the network requiring linear time. We propose a Simhash-based attention scheme which approximates full attention while providing tuneable parameters which trades off runtime for accuracy. Preliminary results suggest that O(n log n) time is sufficient for a good approximation of full attention. Furthermore, our method naturally admits an implementation using O(n) memory, matching the memory cost of other space-efficient attention mechanisms. It has been shown that large-scale diffusion models can be better at memorizing its training data than GANs, the previous state-of-the-art in image generative modeling. This poses significant privacy risks. We combine previous advances in differentially private machine learning and apply it to latent diffusion models (LDMs). Specifically, we use DP-SGD to train a subset of parameters in the attention modules of the model. We make comparisons with existing works using Fréchet inception distance (FID), a perceptual distance metric, and validation-set classification accuracies of downstream classifier models trained on synthetic data.
Degree
thesis:*- Name thesis:degree_name
- Master of Science - MSc
- Level thesis:degree_level
- master's
- Discipline thesis:degree_discipline
- Computer Science
- Grantor dc:publisher
- University of British Columbia
- Year dc:date
- 2024
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Liu, Michael F.
- Contributors dc:contributor
-
- Harvey, Nicholas J. A.
Rights
dc:rights- Statement dc:rights
-
- Attribution 4.0 International
- Language dc:language
- eng
Identifiers
dc:identifier.*- Handle dc:identifier
- http://hdl.handle.net/2429/89134