Back to search

Cornell University

Stochastic Gradient Methods with Bias and Momentum

Abstract

dc:description.abstract

This dissertation investigates stochastic optimization methods where exact gradient computations are prohibitively expensive, necessitating potentially biased gradient estimators. Such challenges arise in applications like empirical risk minimization in machine learning, where unbiased gradient estimates are not always feasible, and in derivative-free settings where only function values are accessible. We focus on stochastic gradient methods with biased estimates from shuffling-based sampling schemes, which lack standard assumptions of independence and unbiasedness. While unbiased methods like i.i.d. SGD are well-studied, the analytical challenges of biased shuffling methods, particularly when combined with momentum, are underexplored. This work addresses these gaps by examining the theoretical and practical impacts of shuffling schemes in stochastic first-order methods, including stochastic gradient descent and its variant with heavy ball momentum and Nesterov's acceleration. Additionally, we analyze a backtracking variant of FISTA and extend a stochastic analysis framework from prior work to both ISTA and backtracking FISTA, showing that, without requiring unbiased gradient estimators, these methods retain their deterministic complexity.

Degree

thesis:*
Name thesis:degree_name
Ph. D., Operations Research and Information Engineering
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Operations Research and Information Engineering
Grantor
Cornell University
Year dc:date.issued
2024

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Tran, Trang
Chair dc:contributor.chair
  • Scheinberg, Katya
Committee members dc:contributor.committeemember
  • Pender, Jamol
  • Henderson, Shane

Subjects

dc:subject × 4

Rights

Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Bibid
16922018
Dc Identifier Other
ProQuest Submission ID: 14752
ProQuest Publication ID: 31641026

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Tran, Trang. Stochastic Gradient Methods with Bias and Momentum. Doctor of Philosophy thesis, Cornell University, 2024. https://hdl.handle.net/1813/117239