This work is concerned with the theoretical study and the implementation of algorithms for solving two particular types of nonlinear optimization problems, namely unconstrained and simple-bound constrained optimization problems. For unconstrained optimization, we develop a new algorithm which uses a filter technique and a trust-region method in order to enforce global convergence and to improve the efficiency of traditional approaches. We also analyze the effect of approximate first and second derivatives on the performance of the filter-trust-region algorithm. We next extend our algorithm to simple-bound constrained optimization problems by combining these ideas with a gradient-projection method. Numerical results follow the proposed methods and indicate that they are competitive with more classical trust-region algorithms.

No description available.

Numerical noise is a prevalent concern in many practical optimization problems. Convergence of gradient based optimization algorithms in the presence of numerical noise is not always assured. One way to improve optimization algorithm performance in the presence of numerical noise is to adjust the method of gradient computation. This study investigates the use of Continuous Sensitivity Equation (CSE) gradient approximations in the context of numerical noise and optimization. Three problems are considered: a problem with a system of ODE constraints, a single parameter flow problem constrained by the Navier-Stokes equations, and a multiple parameter flow problem constrained by the Navier-Stokes equations. All three problems use adaptive methods in the simulation of the constraint and are numerically noisy. Gradients for each problem are computed with both CSE and finite difference methods. The gradients are analyzed and compared. The two flow problems are optimized with a trust region optimization algorithm using both sets of gradient calculations. Optimization results are also compared, and the CSE gradient approximation yields impressive results for these examples. / Master of Science

This work is concerned with the study of nonlinear nonconvex stochastic programming, in particular in the context of trust-region approaches. We first explore how to exploit the structure of multistage stochastic nonlinear programs with linear constraints, in the framework of primal-dual interior point methods. We next study consistency of sample average approximations (SAA) for general nonlinear stochastic programs. We also develop a new algorithm to solve the SAA problem, using the statistical inference information to reduce numercial costs, by means of an internal variable sample size strategy. We finally assess the numerical efficiency of the proposed method for the estimation of discrete choice models, more precisely mixed logit models, using our software AMLET, written for this purpose.

Trust region subproblems arise within a class of unconstrained methods called trust region methods. The subproblems consist of minimizing a quadratic function subject to a norm constraint. This thesis is a survey of different methods developed to find an approximate solution to the subproblem. We study the well-known method of More and Sorensen and two recent methods for large sparse subproblems: the so-called Lanczos method of Gould et al. and the Rendland Wolkowicz algorithm. The common ground to explore these methods will be semidefinite programming. This approach has been used by Rendl and Wolkowicz to explain their method and the More and Sorensen algorithm; we extend this work to the Lanczos method. The last chapter of this thesis is dedicated to some improvements done to the Rendl and Wolkowicz algorithm and to comparisons between the Lanczos method and the Rendl and Wolkowicz algorithm. In particular, we show some weakness of the Lanczos method and show that the Rendl and Wolkowicz algorithm is more robust.

Trust region subproblems arise within a class of unconstrained methods called trust region methods. The subproblems consist of minimizing a quadratic function subject to a norm constraint. This thesis is a survey of different methods developed to find an approximate solution to the subproblem. We study the well-known method of More and Sorensen and two recent methods for large sparse subproblems: the so-called Lanczos method of Gould et al. and the Rendland Wolkowicz algorithm. The common ground to explore these methods will be semidefinite programming. This approach has been used by Rendl and Wolkowicz to explain their method and the More and Sorensen algorithm; we extend this work to the Lanczos method. The last chapter of this thesis is dedicated to some improvements done to the Rendl and Wolkowicz algorithm and to comparisons between the Lanczos method and the Rendl and Wolkowicz algorithm. In particular, we show some weakness of the Lanczos method and show that the Rendl and Wolkowicz algorithm is more robust.

Neural networks optimizers are dominated by first-order methods, due to their inexpensive computational cost per iteration. However, it has been shown that firstorder optimization is prone to reaching sharp minima when trained with large batch sizes. As the batch size increases, the statistical stability of the problem increases, a regime that is well suited for second-order optimization methods. In this thesis, we study a distributed ellipsoidal trust region model for neural networks. We use a block diagonal approximation of the Hessian, assigning consecutive layers of the network to each process. We solve in parallel for the update direction of each subset of the parameters. We show that our optimizer is fit for large batch training as well as increasing number of processes.

In this thesis a numerical study of the one dimensional viscous Burgers equation is conducted. The discretization techniques Finite Differences, Finite Element Method and Group Finite Elements are applied and their impact on model reduction techniques, namely Proper Orthogonal Decomposition (POD), Group POD and the Discrete Empirical Interpolation Method (DEIM), is studied. This study is facilitated by examination of several common ODE solvers. Embedded in this process, some results on the structure of the POD basis and an alternative algorithm to compute the POD subspace are presented. Various numerical studies are conducted to compare the different methods and the to study the interaction of the spatial discretization on the ROM through the basis functions. Moreover, the results are used to investigate the impact of Reduced Order Models (ROM) on Optimal Control Problems. To this end, the ROM is embedded in a Trust Region Framework and the convergence results of Arian et al. (2000) is extended to POD-DEIM. Based on the convergence theorem and the results of the numerical studies, the emphasis is on implementation strategies for numerical speedup. / Master of Science

The research conducted in this dissertation is divided into two main parts. The first part provides further improvements in power system state estimation and the second part implements Contingency Constrained Optimal Power Flow (CCOPF) in a stochastic multiple contingency framework. As a real-time application in modern power systems, the existing Newton-QR state estimation algorithms are too slow and too fragile numerically. This dissertation presents a new and more robust method that is based on trust region techniques. A faster method was found among the class of Krylov subspace iterative methods, a robust implementation of the conjugate gradient method, called the LSQR method. Both algorithms have been tested against the widely used Newton-QR state estimator on the standard IEEE test networks. The trust region method-based state estimator was found to be very reliable under severe conditions (bad data, topological and parameter errors). This enhanced reliability justifies the additional time and computational effort required for its execution. The numerical simulations indicate that the iterative Newton-LSQR method is competitive in robustness with classical direct Newton-QR. The gain in computational efficiency has not come at the cost of solution reliability. The second part of the dissertation combines Sequential Quadratic Programming (SQP)-based CCOPF with Monte Carlo importance sampling to estimate the operating cost of multiple contingencies. We also developed an LP-based formulation for the CCOPF that can efficiently calculate Locational Marginal Prices (LMPs) under multiple contingencies. Based on Monte Carlo importance sampling idea, the proposed algorithm can stochastically assess the impact of multiple contingencies on LMP-congestion prices.

Derivative free optimization algorithms are implementations of trust region based derivative-free methods using multivariate polynomial interpolation. These are designed to minimize smooth functions whose derivatives are not available or costly to compute. The trust region based multilevel optimization algorithms for solving large scale unconstrained optimization problems resulting by discretization of partial differential equations (PDEs), make use of different discretization levels to reduce the computational cost. In this thesis, a derivative free multilevel optimization algorithm is derived and its convergence behavior is analyzed. The effectiveness of the algorithms is demonstrated on a shape optimization problem.