Back to search

Cornell University

Measuring The Unmeasured: New Threats to Machine Learning Systems

Abstract

dc:description.abstract

Machine learning (ML) is at the core of many Internet services and applications. Practitioners evaluate ML models based on the accuracy metrics, which measures the models' predictive power on unseen future data. On the other hand, as ML systems are becoming more personalized and more important in decision-making, malicious adversaries have an incentive to interfere with the ML environment for various purposes such as extracting information about sensitive training data or inducing desired behavior in models' output. However, none of these security and privacy threats are captured by accuracy and it is unclear to what extent current ML systems could go wrong. In this dissertation, we identify and quantify a number of threats to ML systems that are not measured by conventional performance metrics: (1) we consider the privacy threats at training time, where we show that adversary can supply malicious training code to force a ML model into intentionally "memorizing" sensitive training data, and later extract memorized information from the model; (2) motivated by data-protection regulations, we identify a compliance issue where personal information might be collected for training ML models without consent, and design practical auditing techniques for detecting such unauthorized data collection; (3) we study overlearning phenomenon in deep learning models where the internal representations reveal sensitive and uncorrelated information, and discuss its implications in terms of privacy leakages and compliance with regulations; and (4) we demonstrate a secure venerability in ML models for analyzing text semantic similarity, where we propose attacks for generating texts that are semantically unrelated but judged as similar by these ML models. The goal of this dissertation is to provide ML practitioners ways for measuring risks in the ML models through threat modeling. We hope that our proposed attacks could give insights for better mitigation methods, and advocate the ML community to consider all aspects rather than only accuracy when designing new learning algorithms and building new ML systems.

Degree

thesis:*
Name thesis:degree_name
Ph. D., Computer Science
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Computer Science
Grantor
Cornell University
Year dc:date.issued
2020

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Song, Congzheng
Chair dc:contributor.chair
  • Shmatikov, Vitaly
Committee members dc:contributor.committeemember
  • Nissenbaum, Helen
  • Ristenpart, Thomas

Subjects

dc:subject × 3

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Bibid
13312082
Dc Identifier Other
ProQuest Submission ID: 12342
ProQuest Publication ID: 28257500

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Song, Congzheng. Measuring The Unmeasured: New Threats to Machine Learning Systems. Doctor of Philosophy thesis, Cornell University, 2020. https://hdl.handle.net/1813/103372