abstract: This thesis presents a family of adaptive curvature methods for gradient-based stochastic optimization. In particular, a general algorithmic framework is introduced along with a practical implementation that yields an efficient, adaptive curvature gradient descent algorithm. To this end, a theoretical and practical link between curvature matrix estimation and shrinkage methods for covariance matrices is established. The use of shrinkage improves estimation accuracy of the curvature matrix when data samples are scarce. This thesis also introduce several insights that result in data- and computation-efficient update equations. Empirical results suggest that the proposed method compares favorably with existing second-order techniques based on the Fisher or Gauss-Newton and with adaptive stochastic gradient descent methods on both supervised and reinforcement learning tasks. / Dissertation/Thesis / Masters Thesis Computer Science 2019

Thesis (M.S.)--Brigham Young University. Dept. of Mechanical Engineering, 2003. / Includes bibliographical references (p. 87-89).

Thesis (Ph.D.)--University of Tulsa, 1993. / Includes bibliographical references (leaves 254-269).

Thesis (M. Eng.)(Electronics)--University of Pretoria, 2001. / Title from opening screen (viewed March 10, 2005. Summaries in Afrikaans and English. Includes bibliography and index.

Thesis (Ph. D.)--Industrial and Systems Engineering, Georgia Institute of Technology, 2006. / Parker, R. Gary, Committee Member ; Shapiro, Alexander, Committee Member ; Nemirovski, Arkadi, Committee Member ; Green, William, Committee Member ; Monteiro, Renato, Committee Chair.

Orientadores: Aurélio Ribeiro Leite de Oliveira, Marta Ines Velazco Fontova / Dissertação (mestrado) - Universidade Estadual de Campinas, Instituto de Matemática, Estatística e Computação Científica / Made available in DSpace on 2018-08-18T12:49:39Z (GMT). No. of bitstreams: 1 Coelho_AlessandroFonsecaEsteves_M.pdf: 2659631 bytes, checksum: fc1bec925179612ee07a4aaef7092d8a (MD5) Previous issue date: 2011 / Resumo: Um método frequentemente utilizado para a solução de problemas de programação linear é o método de pontos interiores. Nestes métodos precisamos resolver sistemas lineares para calcular a direção de Newton a cada iteração. A solução desses sistemas consiste no passo de maior esforço computacional nos métodos de pontos interiores. A fatoração de Cholesky é a opção mais utilizada para resolver estes sistemas. Contudo, quando trabalhamos com problemas de grande porte, esta fatoração pode ser densa e torna-se inviável trabalhar com esses métodos. Nestes casos, uma boa opção consiste no uso de métodos iterativos precondicionados. Estudos anteriores utilizam o método dos gradientes conjugados precondicionado para obter uma solução destes sistemas. Particularmente, os sistemas originados dos métodos de pontos interiores, são, naturalmente, sistemas de equações normais. Porém, a versão padrão do método dos gradientes conjugados, não considera a estrutura de equações normais do sistema. Neste trabalho propomos a utilização de duas versões do método de gradientes conjugados precondicionado que consideram a estrutura de equações normais destes sistemas. Estas versões serão comparadas com a versão de gradientes conjugados precondicionada que não considera a estrutura de equações normais do sistema. Resultados numéricos com problemas de grande porte mostram que uma dessas versões é competitiva em relação à versão padrão / Abstract: An often used method for solving linear programming problems is the interior point method. In these methods we need to solve linear systems to compute the Newton search direction at each iteration. The solution of these systems is the procedure of most computational effort in interior point methods. The Cholesky factorization is the most often used method to solve these systems. However, when dealing with large scale problems, this factorization can be dense and it become impossible to apply such methods. In such cases, a good option is the use of preconditioned iterative methods. Previous studies have used the preconditioned conjugate gradient method to find the solution of these systems. Particularly, the systems arising from interior point methods are, naturally, systems of normal equations type. Nevertheless, the standard version of the conjugate gradient method, does not take into account the normal equations system structure. This study proposes the use of two versions of preconditioned conjugate gradient method considering the normal equations structure of these systems. These versions are compared with the preconditioned conjugate gradient version that does not consider that structure. Numerical results with large scale problems show that one of these versions is competitive with the standard one / Mestrado / Matematica Aplicada / Mestre em Matemática Aplicada

Non-convex optimization problems appear in various applications of machine learning. Because of their practical importance, these problems gained a lot of attention in recent years, leading to the rapid development of new efficient stochastic gradient-type methods. In the quest to improve the generalization performance of modern deep learning models, practitioners are resorting to using larger and larger datasets in the training process, naturally distributed across a number of edge devices. However, with the increase of trainable data, the computational costs of gradient-type methods increase significantly. In addition, distributed methods almost invariably suffer from the so-called communication bottleneck: the cost of communication of the information necessary for the workers to jointly solve the problem is often very high, and it can be orders of magnitude higher than the cost of computation. This thesis provides a study of first-order stochastic methods addressing these issues. In particular, we structure this study by considering certain classes of methods. That allowed us to understand current theoretical gaps, which we successfully filled by providing new efficient algorithms.

This thesis is broadly concerned with online-optimizing anti-windup control. These are control structures that implement some online-optimization routines to compensate for the windup effects in constrained control systems. The first part of this thesis examines a general framework for analyzing robust preservation in anti-windup control systems. This framework - the robust Kalman conjecture - is defined for the robust Lur’e problem. This part of the thesis verifies this conjecture for first-order plants perturbed by various norm-bounded unstructured uncertainties. Integral quadratic constraint theory is exploited to classify the appropriate stability multipliers required for verification in these cases. The remaining part of the thesis focusses on accelerated gradient methods. In particular, tight complexity-certificates can be obtained for the Nesterov gradient method, which makes it attractive for implementation of online-optimizing anti-windup control. This part of the thesis presents a proposed algorithm that extends the classical Nesterov gradient method by using available secant information. Numerical results demonstrating the efficiency of the proposed algorithm are analysed with the aid of performance profiles. As the objective function becomes more ill-conditioned, the proposed algorithm becomes significantly more efficient than the classical Nesterov gradient method. The improved performance bodes well for online-optimization anti-windup control since ill-conditioning is common place in constrained control systems. In addition, this thesis explores another subcategory of accelerated gradient methods known as Barzilai-Borwein gradient methods. Here, two algorithms that modify the Barzilai-Borwein gradient method are proposed. Global convergence of the proposed algorithms for all convex functions is established by using discrete Lyapunov theorems.

Toute perturbation dans les systèmes linéaires peut gravement dégrader la performance des méthodes itératives lorsque les directions conjuguées sont constituées. Ce problème peut être partiellement résolu par les méthodes du gradient à retards, qui ne garantissent pas la descente de la fonction quadratique, mais peuvent améliorer la convergence par rapport aux méthodes traditionnelles. Les travaux ultérieurs se sont concentrés sur les méthodes du gradient alternées avec deux ou plusieurs types de pas afin d'interrompre le zigzag. Des papiers récents ont suggéré que la révélation d'information de second ordre avec des pas à retards pourrait réduire de manière asymptotique les espaces de recherche dans des dimensions de plus en plus petites. Ceci a conduit aux méthodes du gradient avec alignement dans lesquelles l'étape essentielle et l'étape auxiliaire sont effectuées en alternance. Des expériences numériques ont démontré leur efficacité. Cette thèse considère d'abord des méthodes du gradient efficaces pour résoudre les systèmes linéaires symétriques définis positifs. Nous commençons par étudier une méthode alternée avec la propriété de terminaison finie à deux dimensions. Ensuite, nous déduisons davantage de propriétés spectrales pour les méthodes du gradient traditionnelles. Ces propriétés nous permettent d’élargir la famille de méthodes du gradient avec alignement et d’établir la convergence de nouvelles méthodes. Nous traitons également les itérations de gradient comme un processus peu coûteux intégré aux méthodes de splitting. En particulier, nous abordons le problème de l’estimation de paramètre et suggérons d’utiliser les méthodes du gradient rapide comme solveurs internes à faible précision. Dans le cas parallèle, nous nous concentrons sur les formulations avec retards pour lesquelles il est possible de réduire les coûts de communication. Nous présentons également de nouvelles propriétés et méthodes pour les itérations de gradient s-dimensionnelles. En résumé, cette thèse s'intéresse aux trois sujets interreliés dans lesquelles les itérations de gradient peuvent être utilisées en tant que solveurs efficaces, qu’outils intégrés pour les méthodes de splitting et que solveurs parallèles pour réduire la communication. Des exemples numériques sont présentés à la fin de chaque sujet pour appuyer nos résultats théoriques. / Any perturbation in linear systems may severely degrade the performance of iterative methods when conjugate directions are constructed. This issue can be partially remedied by lagged gradient methods, which does not guarantee descent in the quadratic function but can improve the convergence compared with traditional gradient methods. Later work focused on alternate gradient methods with two or more steplengths in order to break the zigzag pattern. Recent papers suggested that revealing of second-order information along with lagged steps could reduce asymptotically the search spaces in smaller and smaller dimensions. This led to gradient methods with alignment in which essential and auxiliary steps are conducted alternately. Numerical experiments have demonstrated their effectiveness. This dissertation first considers efficient gradient methods for solving symmetric positive definite linear systems. We begin by studying an alternate method with two-dimensional finite termination property. Then we derive more spectral properties for traditional steplengths. These properties allow us to expand the family of gradient methods with alignment and establish the convergence of new methods. We also treat gradient iterations as an inexpensive process embedded in splitting methods. In particular we address the parameter estimation problem and suggest to use fast gradient methods as low-precision inner solvers. For the parallel case we focus on the lagged formulations for which it is possible to reduce communication costs. We also present some new properties and methods for s-dimensional gradient iterations. To sum up, this dissertation is concerned with three inter-related topics in which gradient iterations can be employed as efficient solvers, as embedded tools for splitting methods and as parallel solvers for reducing communication. Numerical examples are presented at the end of each topic to support our theoretical findings.

In this thesis, we develop block-decomposition (BD) methods and variants of accelerated *9gradient methods for large-scale conic programming and convex optimization, respectively. The BD methods, discussed in the first two parts of this thesis, are inexact versions of proximal-point methods applied to two-block-structured inclusion problems. The adaptive accelerated methods, presented in the last part of this thesis, can be viewed as new variants of Nesterov's optimal method. In an effort to improve their practical performance, these methods incorporate important speed-up refinements motivated by theoretical iteration-complexity bounds and our observations from extensive numerical experiments. We provide several benchmarks on various important problem classes to demonstrate the efficiency of the proposed methods compared to the most competitive ones proposed earlier in the literature. In the first part of this thesis, we consider exact BD first-order methods for solving conic semidefinite programming (SDP) problems and the more general problem that minimizes the sum of a convex differentiable function with Lipschitz continuous gradient, and two other proper closed convex (possibly, nonsmooth) functions. More specifically, these problems are reformulated as two-block monotone inclusion problems and exact BD methods, namely the ones that solve both proximal subproblems exactly, are used to solve them. In addition to being able to solve standard form conic SDP problems, the latter approach is also able to directly solve specially structured non-standard form conic programming problems without the need to add additional variables and/or constraints to bring them into standard form. Several ingredients are introduced to speed-up the BD methods in their pure form such as: adaptive (aggressive) choices of stepsizes for performing the extragradient step; and dynamic updates of scaled inner products to balance the blocks. Finally, computational results on several classes of SDPs are presented showing that the exact BD methods outperform the three most competitive codes for solving large-scale conic semidefinite programming. In the second part of this thesis, we present an inexact BD first-order method for solving standard form conic SDP problems which avoids computations of exact projections onto the manifold defined by the affine constraints and, as a result, is able to handle extra large-scale SDP instances. In this BD method, while the proximal subproblem corresponding to the first block is solved exactly, the one corresponding to the second block is solved inexactly in order to avoid finding the exact solution of a linear system corresponding to the manifolds consisting of both the primal and dual affine feasibility constraints. Our implementation uses the conjugate gradient method applied to a reduced positive definite dual linear system to obtain inexact solutions of the latter augmented primal-dual linear system. In addition, the inexact BD method incorporates a new dynamic scaling scheme that uses two scaling factors to balance three inclusions comprising the optimality conditions of the conic SDP. Finally, we present computational results showing the efficiency of our method for solving various extra large SDP instances, several of which cannot be solved by other existing methods, including some with at least two million constraints and/or fifty million non-zero coefficients in the affine constraints. In the last part of this thesis, we consider an adaptive accelerated gradient method for a general class of convex optimization problems. More specifically, we present a new accelerated variant of Nesterov's optimal method in which certain acceleration parameters are adaptively (and aggressively) chosen so as to: preserve the theoretical iteration-complexity of the original method; and substantially improve its practical performance in comparison to the other existing variants. Computational results are presented to demonstrate that the proposed adaptive accelerated method performs quite well compared to other variants proposed earlier in the literature.