Back to search

Cornell University

PROVABLE AND PRACTICAL ALGORITHMS FOR NON-CONVEX PROBLEMS IN MACHINE LEARNING

Abstract

dc:description.abstract

Machine learning has become one of the most exciting research areas in the world, with various applications. However, there exists a noticeable gap between theory and practice. On one hand, a simple algorithm like stochastic gradient descent (SGD) works very well in practice, without satisfactory theoretical explanations. On the other hand, the algorithms analyzed in the theoretical machine learning literature, although with solid guarantees, tend to be less efficient compared with the techniques widely used in practice, which are usually hand tuned or ad hoc based on intuition. This dissertation is about bridging the gap between theory and practice from two directions. The first direction is "practice to theory", i.e., to explain and analyze the existing algorithms and empirical observations in machine learning. Along this direction, we provide sufficient conditions for SGD to escape saddle points and local minima, as well as SGD dynamics analysis for the two-layer neural network with ReLU activation. The other direction is "theory to practice", i.e., using theoretical tools to obtain new, better and practical algorithms. Along this direction, we introduce a new algorithm Harmonica that uses Fourier analysis and compressed sensing for tuning hyperparameters. Harmonica supports parallel sampling and works well for tuning neural networks with more than 30 hyperparameters.

Degree

thesis:*
Name thesis:degree_name
Ph. D., Computer Science
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Computer Science
Grantor
Cornell University
Year dc:date.issued
2018

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Yuan, Yang
Chair dc:contributor.chair
  • Kleinberg, Robert David
Committee members dc:contributor.committeemember
  • Kozen, Dexter Campbell
  • Joachims, Thorsten

Subjects

dc:subject × 8

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Dc Identifier Bibid
10489357
Dc Identifier Other
ProQuest Submission ID: 10809
ProQuest Publication ID: 10814646

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Yuan, Yang. PROVABLE AND PRACTICAL ALGORITHMS FOR NON-CONVEX PROBLEMS IN MACHINE LEARNING. Doctor of Philosophy thesis, Cornell University, 2018. https://hdl.handle.net/1813/59273