Global ETD Search
Search theses and dissertations gathered from participating repositories worldwide. Every result links back to the library that holds it. No account is needed.
Results
Showing 1 to 20 of 278 for “"Stochastic gradient descent"”.
-
On variants of stochastic gradient descent
Stochastic Gradient Descent (SGD) has played a crucial role in the success of modern machine learning methods. The popularity of SGD arises due to its ease of implementation, low memory and computational requirements, and applicability to a wide variety of optimization problems. However, SGD …
-
Interpolation, growth conditions, and stochastic gradient descent
… interpolate, the training data. In such cases, stochastic gradient descent has been shown to converge without decreasing step-sizes or averaging, and can achieve the fast convergence of deterministic gradient methods. Recent work has further shown that stochastic acceleration and line-search …
-
Certain Systems Arising In Stochastic Gradient Descent
Stochastic approximations is a rich branch of probability theory and has a wide range of application. Here we study stochastic approximations from the perspective of gradient descent. An important question is what is the asymptotic limit of a stochastic approximation. In that spirit we will provide …
-
Adaptive Step Sizes for Stochastic Gradient Descent
Stochastic optimization problems arise in various applications. In recent years, the rapid development of machine learning techniques, and the associated training of artificial intelligence, has further increased interest in such problems. A widely used and conceptually very simple algorithm for …
-
Simple Termination Criteria for Stochastic Gradient Descent Algorithm
Stochastic gradient descent (SGD) algorithm is widely used in modern mathematical optimization. Because of its scalability and ease of implementation, SGD is usually preferred to other methods including the gradient descent algorithm in the large scale optimization. Similar to other iterative …
-
Hybrid Distributed Stochastic Gradient Descent for Federated Learning
… this thesis, we propose the Hybrid Distributed Stochastic Gradient Descent (Hybrid DSGD), a training scheme for federated learning which utilizes the advantages of both digital and analog transmissions to reduce communication overhead and latency. We demonstrate why the conventional analog-based …
-
Applications of Stochastic Gradient Descent to Nonnegative Matrix Factorization
We consider the application of stochastic gradient descent (SGD) to the nonnegative matrix factorization (NMF) problem and the unconstrained low-rank matrix factorization problem. While the literature on the SGD algorithm is rich, the application of this specific algorithm to the field of matrix …
-
Theoretical Analysis of Stochastic Gradient Descent in Stochastic Optimization
Stochastic Gradient Descent (SGD) type algorithms have been widely applied to many stochastic optimization problems, such as machine learning. Despite its empirical success, there is still a lack of theoretical understanding of convergence properties of SGD and its variants. The major bottleneck …
-
Aplikace metody Mean Normalized Stochastic Gradient Descent pro rozpoznávání řeči
… optimalizačních technik je mean-normalized stochastic gradient descent, který navrhli Wiesler a spol. [1]. Tato práce dále vysvětluje a zkoumá tuto metodu na problému klasifikace fonémů. Ne všechny závěry Wieslera a spol. byly potvrzeny. Mean-normalized SGD je vhodné použít pouze pokud je …
-
RISK CONVERGENCE AND ALGORITHMIC REGULARIZATION OF DISCRETE-STEPSIZE (STOCHASTIC) GRADIENT DESCENT
… delves into the theoretical analysis of gradient descent (GD) and stochastic gradient descent (SGD), the fundamental algorithms for optimizing machine learning models, particularly in the context of deep learning. While these algorithms have demonstrated remarkable practical success, …
-
Understanding the role of averaging in non-smooth stochastic gradient descent
… differentiable. We prove that after T steps of stochastic gradient descent (SGD), the error of the final iterate is O(logT⁄T) with high probability. We also construct a function from this class for which the error of the final iterate of deterministic gradient descent is Ω(logT⁄T). This shows …
-
Stochastic Gradient Descent For Modern Machine Learning: Theory, Algorithms And Applications
… powered by the seemingly simple and lightweight stochastic gradient method. Variants of the stochastic gradient method (based on iterate averaging) are known to be asymptotically optimal (in terms of predictive performance). This thesis examines non-asymptotic issues surrounding the use of …
-
On some deterministic properties of stochastic gradient descent and random projection sequences
… the convergence rate of sequences generated by stochastic gradient and random projection methods in both finite and infinite-dimensional Hilbert spaces. We begin by revisiting classical stochastic gradient descent methods and show that the sequence of iterates remains bounded under the mild …
-
Parameter estimation and inference of spatial autoregressive model by stochastic gradient descent
Stochastic gradient descent (SGD) is a popular iterative method for model parameter estimation in large-scale data and online learning settings since it goes through the data in only one pass. While SGD has been well studied for independent data, its application to spatially-correlated data largely …
-
High-Dimensional Scaling Limits of Online Stochastic Gradient Descent in Single-Index Models
We analyze the scaling limits of stochastic gradient descent (SGD) with a constant step size in the high-dimensional regime in single-index models. Specifically, we prove limit theorems for the trajectories of finite-dimensional summary statistics of SGD as the dimension tends to infinity. These …
-
Asymptotic Analysis of an Adaptive Stochastic Gradient Descent: Non-convexity and Markovian Dynamics
… a modified AdaGrad-Norm stepsize scheme for stochastic gradient descent based on the AdaGrad stepsize scheme that generalises the scaling factor. An asymptotic analysis of this stepsize scheme is undertaken in the setting of Markovian dynamics with a non-convex objective function. Under …
-
Stability of Random Dynamical Systems in Noisy Oscillators with Shear and in Stochastic Gradient Descent
… in the literature . We then study a class of stochastic differential equations that allows for stronger shear than in the Stuart-Landau oscillator. This strong shear can cause the system to no longer admit a random attractor or to even lose strong completeness, i.e., to no longer admit a …
-
Asynchronous Parallel Stochastic Gradient Descent: A study of the influence of synchronization methods and hyperparameters
… algorithm for training ANNs is often Stochastic Gradient Descent (SGD). SGD is a first-order numerical optimization algorithm that repeatedly takes a step in the negative gradient direction of a loss function. Training ANNs generally require a large amount of data, and with an …
-
Stationary probability distributions of stochastic gradient descent and the success and failure of the diffusion approximation
In this thesis, Stochastic Gradient Descent (SGD), an optimization method originally popular due to its computational efficiency, is analyzed using Markov chain methods. We compute both numerically, and in some cases analytically, the stationary probability distributions (invariant measures) for …
Page 1 of 14