University of British Columbia
Differentially private neural tangent kernels for privacy-preserving data generation and distillation
Abstract
dc:descriptionWith the increasing interest in Deep Learning, data safety issues have become more prevalent as we rely more on Artificial Intelligence. Adversaries can easily obtain sensitive information through various attacks, this dramatically discourages patients and clients from contributing invaluable data that may be beneficial to research. This problem facilitates the need for a gold standard privacy notion. In recent years, Differential Privacy (DP) has been recognized as a gold standard notion of privacy. Among the current popular DP methods, Maximum mean discrepancy (MMD) is a particularly useful distance metric for differentially private data generation. When used with finite-dimensional features it allows us to summarize and privatize the data distribution once, which we can repeatedly use during generator training without further privacy loss. An important question in this framework is, then, what features are useful to distinguish between real and synthetic data distributions, and whether those enable us to generate quality synthetic data. This work considers using the features of neural tangent kernels (NTKs), more precisely empirical NTKs (e-NTKs). We find that, perhaps surprisingly, the expressiveness of the untrained e-NTK features is comparable to that of the features taken from pre-trained perceptual features using public data. As a result, our method improves the privacy-accuracy trade-off compared to other state-of-the-art methods, without relying on any public data, as demonstrated on several tabular and image benchmark datasets. In addition, we also extend NTK to Data Distillation (DD) in federated learning (FL) settings, where we aim to condense sensitive information into a small set of images for deep learning training in a DP manner, we show that our method obtains meaningful results even under class imbalance and spuriously correlated image datasets.
Degree
thesis:*- Name thesis:degree_name
- Master of Science - MSc
- Level thesis:degree_level
- master's
- Discipline thesis:degree_discipline
- Computer Science
- Grantor dc:publisher
- University of British Columbia
- Year dc:date
- 2024
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Yang, Yilin
- Contributors dc:contributor
-
- Park, Mi Jung
- Li, Xiaoxiao
Rights
dc:rights- Statement dc:rights
-
- Attribution-NonCommercial-NoDerivatives 4.0 International
- Language dc:language
- eng
Identifiers
dc:identifier.*- Handle dc:identifier
- http://hdl.handle.net/2429/87742