{"id":{"repo_id":"msu","oai_identifier":"oai:d.lib.msu.edu:etd_52569"},"canonical_url":"https://search.ndltd.org/etd/msu/oai:d.lib.msu.edu:etd_52569","repository":{"repo_id":"msu","name":"Michigan State University","base_url":"https://d.lib.msu.edu/oai"},"display":{"title":"Implicit Regularization of hyperparameters in deep learning : Beyond convexity and small steps","abstract":"Thesis (Ph.D.)--Michigan State University. Computational Mathematics, Science and Engineering - Doctor of Philosophy, 2025","abstract_html":"Thesis (Ph.D.)--Michigan State University. Computational Mathematics, Science and Engineering - Doctor of Philosophy, 2025","abstract_has_math":false,"creators":["Ghosh, Avrajit"],"institution":null,"degree_name":null,"degree_level":null,"degree_discipline":null,"degree_department":null,"school":null,"contributors":["Ravishankar, Saiprasad","Wang, Rongrong","Wright, John","Liu, Sijia"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2025,"date_issued":"2025","date_published":"2025","updated_at":"2026-11-18T18:59:06Z","subjects":["Computer science","Artificial intelligence"],"languages":["eng"],"rights":["In Copyright"],"rights_urls":[],"identifier_entries":[{"key":"dc:identifier","label":"Identifier","values":["etd:52569","local:Ghosh_grad.msu_0128D_21508"],"render_values":[{"text":"etd:52569","href":null,"code":true},{"text":"local:Ghosh_grad.msu_0128D_21508","href":null,"code":true}]}],"person_identifier_entries":[]},"links":{"outbound_url":"https://doi.org/doi:10.25335/pg4a-7q28","outbound_label":"DOI","outbound_source":"dc:identifier","record_url":"https://doi.org/doi:10.25335/pg4a-7q28","record_label":"Repository record","record_kind":"doi","record_source":"dc:identifier","download_url":null,"download_label":null,"download_source":null,"access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://d.lib.msu.edu/oai?verb=GetRecord&metadataPrefix=oai_dc&identifier=oai%3Ad.lib.msu.edu%3Aetd_52569","prefix":"oai_dc"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Ravishankar, Saiprasad","Wang, Rongrong","Wright, John","Liu, Sijia"]},{"key":"dc:creator","label":"Author","values":["Ghosh, Avrajit"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2025"]},{"key":"dc:relation","label":"Dc Relation","values":["Electronic Theses & Dissertations"]},{"key":"dc:type","label":"Dc Type","values":["text","Electronic dissertations"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Computer science","Artificial intelligence"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["eng"]},{"key":"dc:rights","label":"Dc Rights","values":["In Copyright"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["etd:52569","local:Ghosh_grad.msu_0128D_21508","https://doi.org/doi:10.25335/pg4a-7q28"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Thesis (Ph.D.)--Michigan State University. Computational Mathematics, Science and Engineering - Doctor of Philosophy, 2025","Understanding the optimization of gradient-based methods is crucial to comprehending deep learning. This thesis investigates the implicit regularization induced by gradient descent algorithms, focusing on the role of three critical hyperparameters: step-size, momentum, and stochastic noise variance. Departing from past literature that often relies on simplifying assumptions such as convexity or infinitesimal step-sizes, our work concentrates on the practical deep learning regime where these assumptions are often violated. We demonstrate how these hyperparameters implicitly guide the iterate dynamics toward favorable solutions.In the first chapter, we examine the effect of a large step-size in an overparameterized deep matrix factorization problem designed to fit a low-rank matrix. Through a fine-grained analysis, we show that a learning rate beyond a specific threshold, dependent on the depth and singular values of target low rank matrix, drives the iterates toward the flattest global minimum and induces stable oscillations around it. This reveals the ability of gradient descent to escape sharper global minima and stably oscillate about balanced solution. The second chapter investigates the implicit regularization induced by Polyak's heavy-ball momentum. We show that the discrete momentum update approximates a continuous trajectory governed by a modified loss function, which contains an implicit gradient regularizer. Consequently, a finite step-size and momentum jointly steer the iterate trajectory along regions of smaller gradient norm. This analysis is extended to the stochastic version of gradient descent with momentum, where the implicit regularization is shown to depend jointly on the step-size, momentum, and minibatch variance. In the third chapter, we study the implicit regularization of weight-perturbed gradient descent, where perturbations are drawn from a Gaussian distribution before each gradient step. We calculate a stability threshold for this method based on a local quadratic approximation of the loss and show that deep neural networks trained with weight perturbation typically operate near this threshold. Our analysis provides insight into why weight perturbation may improve generalization over full-batch gradient descent and clarifies the roles of noise variance and the number of Monte Carlo samples in this process. The fourth chapter explores the implicit architectural bias of deep convolutional neural networks in unsupervised image reconstruction. We show that while dense architectures are prone to overfitting measurement noise, sparse architectures are resilient and can recover clean images without the need for early stopping. We propose an algorithm called \"Optimal Eye Surgeon\" to efficiently recover these sparse architectures and train them to alleviate overfitting. The fifth chapter assesses the efficacy of explicit regularization for signal reconstruction in inverse problems. We demonstrate that these explicit regularizers can be learned within an $\\ell_{1}$ minimization framework through bilevel optimization. This chapter complements the rest of the thesis by showing the importance of external regularization in solving inverse problems.In the final chapter, we explore promising future directions for understanding deep learning phenomena through the lens of implicit regularization in optimization.","Description based on online resource. Title from PDF t.p. (Michigan State University Fedora Repository, viewed ).","Includes bibliographical references."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf","1 online resource (286 p.) : ill."]},{"key":"dc:title","label":"Title","values":["Implicit Regularization of hyperparameters in deep learning : Beyond convexity and small steps"]}]}],"canonical_facts":{"dc:contributor":["Ravishankar, Saiprasad","Wang, Rongrong","Wright, John","Liu, Sijia"],"dc:creator":["Ghosh, Avrajit"],"dc:date":["2025"],"dc:description":["Thesis (Ph.D.)--Michigan State University. Computational Mathematics, Science and Engineering - Doctor of Philosophy, 2025","Understanding the optimization of gradient-based methods is crucial to comprehending deep learning. This thesis investigates the implicit regularization induced by gradient descent algorithms, focusing on the role of three critical hyperparameters: step-size, momentum, and stochastic noise variance. Departing from past literature that often relies on simplifying assumptions such as convexity or infinitesimal step-sizes, our work concentrates on the practical deep learning regime where these assumptions are often violated. We demonstrate how these hyperparameters implicitly guide the iterate dynamics toward favorable solutions.In the first chapter, we examine the effect of a large step-size in an overparameterized deep matrix factorization problem designed to fit a low-rank matrix. Through a fine-grained analysis, we show that a learning rate beyond a specific threshold, dependent on the depth and singular values of target low rank matrix, drives the iterates toward the flattest global minimum and induces stable oscillations around it. This reveals the ability of gradient descent to escape sharper global minima and stably oscillate about balanced solution. The second chapter investigates the implicit regularization induced by Polyak's heavy-ball momentum. We show that the discrete momentum update approximates a continuous trajectory governed by a modified loss function, which contains an implicit gradient regularizer. Consequently, a finite step-size and momentum jointly steer the iterate trajectory along regions of smaller gradient norm. This analysis is extended to the stochastic version of gradient descent with momentum, where the implicit regularization is shown to depend jointly on the step-size, momentum, and minibatch variance. In the third chapter, we study the implicit regularization of weight-perturbed gradient descent, where perturbations are drawn from a Gaussian distribution before each gradient step. We calculate a stability threshold for this method based on a local quadratic approximation of the loss and show that deep neural networks trained with weight perturbation typically operate near this threshold. Our analysis provides insight into why weight perturbation may improve generalization over full-batch gradient descent and clarifies the roles of noise variance and the number of Monte Carlo samples in this process. The fourth chapter explores the implicit architectural bias of deep convolutional neural networks in unsupervised image reconstruction. We show that while dense architectures are prone to overfitting measurement noise, sparse architectures are resilient and can recover clean images without the need for early stopping. We propose an algorithm called \"Optimal Eye Surgeon\" to efficiently recover these sparse architectures and train them to alleviate overfitting. The fifth chapter assesses the efficacy of explicit regularization for signal reconstruction in inverse problems. We demonstrate that these explicit regularizers can be learned within an $\\ell_{1}$ minimization framework through bilevel optimization. This chapter complements the rest of the thesis by showing the importance of external regularization in solving inverse problems.In the final chapter, we explore promising future directions for understanding deep learning phenomena through the lens of implicit regularization in optimization.","Description based on online resource. Title from PDF t.p. (Michigan State University Fedora Repository, viewed ).","Includes bibliographical references."],"dc:format":["application/pdf","1 online resource (286 p.) : ill."],"dc:identifier":["etd:52569","local:Ghosh_grad.msu_0128D_21508","https://doi.org/doi:10.25335/pg4a-7q28"],"dc:language":["eng"],"dc:relation":["Electronic Theses & Dissertations"],"dc:rights":["In Copyright"],"dc:subject":["Computer science","Artificial intelligence"],"dc:title":["Implicit Regularization of hyperparameters in deep learning : Beyond convexity and small steps"],"dc:type":["text","Electronic dissertations"]},"updated_at":"2026-11-18T18:59:06Z"}