Thesis: Ph. D., Massachusetts Institute of Technology, Department of Brain and Cognitive Sciences, 2019 / Cataloged from PDF version of thesis. / Includes bibliographical references (pages 165-186). / Non-convex optimization and learning play an important role in data science and machine learning, yet so far they still elude our understanding in many aspects. In this thesis, I study two important aspects of non-convex optimization and learning: Riemannian optimization and deep neural networks. In the first part, I develop iteration complexity analysis for Riemannian optimization, i.e., optimization problems defined on Riemannian manifolds. Through bounding the distortion introduced by the metric curvature, iteration complexity of Riemannian (stochastic) gradient descent methods is derived. I also show that some fast first-order methods in Euclidean space, such as Nesterov's accelerated gradient descent (AGD) and stochastic variance reduced gradient (SVRG), have Riemannian counterparts that are also fast under certain conditions. In the second part, I challenge two common practices in deep learning, namely empirical risk minimization (ERM) and normalization. Specifically, I show (1) training on convex combinations of samples improves model robustness and generalization, and (2) a good initialization is sufficient for training deep residual networks without normalization. The method in (1), called mixup, is motivated by a data-dependent Lipschitzness regularization of the network. The method in (2), called Zerolnit, makes the network update scale invariant to its depth at initialization. / by Hongyi Zhang. / Ph. D. / Ph.D. Massachusetts Institute of Technology, Department of Brain and Cognitive Sciences
| Identifer | oai:union.ndltd.org:MIT/oai:dspace.mit.edu:1721.1/121830 |
| Date | January 2019 |
| Creators | Zhang, Hongyi,Ph. D.Massachusetts Institute of Technology. |
| Contributors | Suvrit Sra., Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences., Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences |
| Publisher | Massachusetts Institute of Technology |
| Source Sets | M.I.T. Theses and Dissertation |
| Language | English |
| Detected Language | English |
| Type | Thesis |
| Format | 186 pages, application/pdf |
| Rights | MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission., http://dspace.mit.edu/handle/1721.1/7582 |
Page generated in 0.0019 seconds