Denna kandidatrapport studerar en projektuppgift som har utförts av en grupp studenter på Linköpings universitet. Uppgiften har givits av en industridoktorand på Saab och härstammar från reglering av styrsystem i stridsflygplan. Det som har tagits fram av kandidatgruppen är en kvadratisk optimeringslösare som även kan köras från MATLAB. Undersökningar har gjorts om det går att implementera optimeringsalgoritmen i programspråket C med projektets tidsbegränsning, om lösaren kan bli lika snabb som den kommersiella produkten Gurobi och om projektet går att utföra utan någon speciell utvecklingsmetodik. I resultat går det att se att det gick att implementera optimeringsalgoritmen, att lösaren inte kunde bli lika snabb som Gurobi och att kandidatgruppen inte använde någon speciell utvecklingsmetodik. Slutsatser kandidatgruppen har dragit ar att valet av optimeringsalgoritm inte var helt genomtänkt, att mer mer tid och resurser hade lösaren kanske kunnat blivit lika snabb som Gurobi och att arbetet fungerade tillfredsställande utan någon speciell utvecklingsmetod.

In model predictive control an optimization problem is solved in every time step, which in real-time applications has to be solved within a limited time frame. When applied on embedded hardware in fast changing systems it is important to use efficient solvers and crucial to guarantee that the optimization problem can be solved within the time frame. In this thesis a path following controller which follows a motion plan given by a motion planner is implemented to steer a truck and trailer system. To solve the optimization problems which in this thesis are quadratic programs the three different solvers DAQP, qpOASES and OSQP are employed. The computational time of the active-set solvers DAQP, qpOASES and the operator splitting solver OSQP are compared, where the controller using DAQP was found the fastest and therefore most suited to use in this application of real-time model predictive control.  A certification framework for the active-set method is used to give complexity guarantees on the controller using DAQP. The exact worst-case number of iterations when the truck and trailer system is following a straight path is presented. Furthermore, initial experiments show that given enough computational time/power the exact iteration complexity can be determined for every possible quadratic program that can appear in the controller.

The Support Vector Machine (SVM) is a popular binary classification model due to its superior generalization performance, relative ease-of-use, and applicability of kernel methods. SVM training entails solving an associated quadratic programming (QP) that presents significant challenges in terms of speed and memory constraints for very large datasets; therefore, research on numerical optimization techniques tailored to SVM training is vast. Slow training times are especially of concern when one considers that re-training is often necessary at several values of the models regularization parameter, C, as well as associated kernel parameters. The active set method is suitable for solving SVM problem and is in general ideal when the Hessian is dense and the solution is sparse–the case for the `1-loss SVM formulation. There has recently been renewed interest in the active set method as a technique for exploring the entire SVM regularization path, which has been shown to solve the SVM solution at all points along the regularization path (all values of C) in not much more time than it takes, on average, to perform training at a single value of C with traditional methods. Unfortunately, the majority of active set implementations used for SVM training require positive definite kernels, and those implementations that do allow semi-definite kernels tend to be complex and can exhibit instability and, worse, lack of convergence. This severely limits applicability since it precludes the use of the linear kernel, can be an issue when duplicate data points exist, and doesn’t allow use of low-rank kernel approximations to improve tractability for large datasets. The difficulty, in the case of a semi-definite kernel, arises when a particular active set results in a singular KKT matrix (or the equality-constrained problem formed using the active set is semidefinite). Typically this is handled by explicitly detecting the rank of the KKT matrix. Unfortunately, this adds significant complexity to the implementation; and, if care is not taken, numerical iii instability, or worse, failure to converge can result. This research shows that the singular KKT system can be avoided altogether with simple modifications to the active set method. The result is a practical, easy to implement active set method that does not need to explicitly detect the rank of the KKT matrix nor modify factorization or solution methods based upon the rank. Methods are given for both conventional SVM training as well as for computing the regularization path that are simple and numerically stable. First, an efficient revised simplex method is efficiently implemented for SVM training (SVM-RSQP) with semi-definite kernels and shown to out-perform competing active set implementations for SVM training in terms of training time as well as shown to perform on-par with state-of-the-art SVM training algorithms such as SMO and SVMLight. Next, a new regularization path-following algorithm for semi-definite kernels (Simple SVMPath) is shown to be orders of magnitude faster, more accurate, and significantly less complex than competing methods and does not require the use of external solvers. Theoretical analysis reveals new insights into the nature of the path-following algorithms. Finally, a method is given for computing the approximate regularization path and approximate kernel path using the warm-start capability of the proposed revised simplex method (SVM-RSQP) and shown to provide significant, orders of magnitude, speed-ups relative to the traditional grid search where re-training is performed at each parameter value. Surprisingly, it also shown that even when the solution for the entire path is not desired, computing the approximate path can be seen as a speed-up mechanism for obtaining the solution at a single value. New insights are given concerning the limiting behaviors of the regularization and kernel path as well as the use of low-rank kernel approximations.

<p>The focus of this thesis is on numerical algorithms for efficient solution of QTL analysis problem in genetics.</p><p>Firstly, we consider QTL mapping problems where a standard least-squares model is used for computing the model fit. We develop optimization methods for the local problems in a hybrid global-local optimization scheme for determining the optimal set of QTL locations. Here, the local problems have constant bound constraints and may be non-convex and/or flat in one or more directions. We propose an enhanced quasi-Newton method and also implement several schemes for constrained optimization. The algorithms are adopted to the QTL optimization problems. We show that it is possible to use the new schemes to solve problems with up to 6 QTLs efficiently and accurately, and that the work is reduced with up to two orders magnitude compared to using only global optimization.</p><p>Secondly, we study numerical methods for QTL mapping where variance component estimation and a REML model is used. This results in a non-linear optimization problem for computing the model fit in each set of QTL locations. Here, we compare different optimization schemes and adopt them for the specifics of the problem. The results show that our version of the active set method is efficient and robust, which is not the case for methods used earlier. We also study the matrix operations performed inside the optimization loop, and develop more efficient algorithms for the REML computations. We develop a scheme for reducing the number of objective function evaluations, and we accelerate the computations of the derivatives of the log-likelihood by introducing an efficient scheme for computing the inverse of the variance-covariance matrix and other components of the derivatives of the log-likelihood.</p>

The focus of this thesis is on numerical algorithms for efficient solution of QTL analysis problem in genetics. Firstly, we consider QTL mapping problems where a standard least-squares model is used for computing the model fit. We develop optimization methods for the local problems in a hybrid global-local optimization scheme for determining the optimal set of QTL locations. Here, the local problems have constant bound constraints and may be non-convex and/or flat in one or more directions. We propose an enhanced quasi-Newton method and also implement several schemes for constrained optimization. The algorithms are adopted to the QTL optimization problems. We show that it is possible to use the new schemes to solve problems with up to 6 QTLs efficiently and accurately, and that the work is reduced with up to two orders magnitude compared to using only global optimization. Secondly, we study numerical methods for QTL mapping where variance component estimation and a REML model is used. This results in a non-linear optimization problem for computing the model fit in each set of QTL locations. Here, we compare different optimization schemes and adopt them for the specifics of the problem. The results show that our version of the active set method is efficient and robust, which is not the case for methods used earlier. We also study the matrix operations performed inside the optimization loop, and develop more efficient algorithms for the REML computations. We develop a scheme for reducing the number of objective function evaluations, and we accelerate the computations of the derivatives of the log-likelihood by introducing an efficient scheme for computing the inverse of the variance-covariance matrix and other components of the derivatives of the log-likelihood.

Nonnegative matrix factorization (NMF) is a useful dimension reduction method that has been investigated and applied in various areas. NMF is considered for high-dimensional data in which each element has a nonnegative value, and it provides a low-rank approximation formed by factors whose elements are also nonnegative. The nonnegativity constraints imposed on the low-rank factors not only enable natural interpretation but also reveal the hidden structure of data. Extending the benefits of NMF to multidimensional arrays, nonnegative tensor factorization (NTF) has been shown to be successful in analyzing complicated data sets. Despite the success, NMF and NTF have been actively developed only in the recent decade, and algorithmic strategies for computing NMF and NTF have not been fully studied. In this thesis, computational challenges regarding NMF, NTF, and related least squares problems are addressed. First, efficient algorithms of NMF and NTF are investigated based on a connection from the NMF and the NTF problems to the nonnegativity-constrained least squares (NLS) problems. A key strategy is to observe typical structure of the NLS problems arising in the NMF and the NTF computation and design a fast algorithm utilizing the structure. We propose an accelerated block principal pivoting method to solve the NLS problems, thereby significantly speeding up the NMF and NTF computation. Implementation results with synthetic and real-world data sets validate the efficiency of the proposed method. In addition, a theoretical result on the classical active-set method for rank-deficient NLS problems is presented. Although the block principal pivoting method appears generally more efficient than the active-set method for the NLS problems, it is not applicable for rank-deficient cases. We show that the active-set method with a proper starting vector can actually solve the rank-deficient NLS problems without ever running into rank-deficient least squares problems during iterations. Going beyond the NLS problems, it is presented that a block principal pivoting strategy can also be applied to the l1-regularized linear regression. The l1-regularized linear regression, also known as the Lasso, has been very popular due to its ability to promote sparse solutions. Solving this problem is difficult because the l1-regularization term is not differentiable. A block principal pivoting method and its variant, which overcome a limitation of previous active-set methods, are proposed for this problem with successful experimental results. Finally, a group-sparsity regularization method for NMF is presented. A recent challenge in data analysis for science and engineering is that data are often represented in a structured way. In particular, many data mining tasks have to deal with group-structured prior information, where features or data items are organized into groups. Motivated by an observation that features or data items that belong to a group are expected to share the same sparsity pattern in their latent factor representations, We propose mixed-norm regularization to promote group-level sparsity. Efficient convex optimization methods for dealing with the regularization terms are presented along with computational comparisons between them. Application examples of the proposed method in factor recovery, semi-supervised clustering, and multilingual text analysis are presented.