Optimization is an important part of mathematics and is mostly used for practical applications. For specific types of objective functions, a lot of different methods exist. A method to use when the objective is unknown and/or expensive can be difficult to determine. One of the answers is bayesian optimization, which instead of direct optimization creates a probabilistic model and uses it to constructs easily optimizable auxiliary function. It is an iterative method that uses information from previous iterations to find new point in which the objective is evaluated and tries to find the optimum within a fewer iterations. This thesis introduces bayesian optimization, suma- rizes its different approaches in lower and higher dimensions and shows when to use it suitably. An important part of the thesis is my own optimization algorithm which is applied to different practical problems - e.g. parameter optimization in machine learning algorithm. 1

Ce mémoire par article part de la question suivante: pouvons-nous utiliser des prothèses neurales afin d’activer artificiellement certain muscles dans le but d’accélérer la guérison et le réapprentissage du contrôle moteur après un AVC ou un traumatisme cervical ? Cette question touche plus de 15 millions de personnes chaque année à travers le monde, et est au coeur de la recherche de Numa Dancause et Marco Bonizzato, nos collaborateurs dans le département de Neuroscience de l’Université de Montréal. Il est maintenant possible d’implanter des électrodes à grande capacité dans le cortex dans le but d’acheminer des signaux électriques, mais encore difficile de prédire l’effet de stimulations sur le cerveau et le reste du corps. Cependant, des résultats préliminaires prometteurs sur des rats et singes démontrent qu’une récupération motrice non-négligeable est observée après stimulation de régions encore fonctionnelles du cortex moteur. Les difficultés rattachées à l’implémentation optimale de stimulation motocorticale consistent donc à trouver une de ces régions, ainsi qu’un protocole de stimulation efficace à la récupération. Bien que cette optimisation a été jusqu’à présent faite à la main, l’émergence d’implants capables de livrer des signaux sur plusieurs sites et avec plusieurs patrons spatio-temporels rendent l’exploration manuelle et exhaustive impossible. Une approche prometteuse afin d’automatiser et optimiser ce processus est d’utiliser un algorithme d’exploration bayésienne. Mon travail a été de déveloper et de raffiner ces techniques avec comme objectif de répondre aux deux questions scientifiques importantes suivantes: (1) comment évoquer des mouvements complexes en enchainant des microstimulations corticales ?, et (2) peuvent-elles avoir des effets plus significatifs que des stimulations simples sur la récupération motrice? Nous présentons dans l’article de ce mémoire notre approche hiérarchique utilisant des processus gaussiens pour exploiter les propriétés connues du cerveau afin d’accélérer la recherche, ainsi que nos premiers résultats répondant à la question 1. Nous laissons pour des travaux futur une réponse définitive à la deuxième question. / The idea for this thesis by article sprung from the following question: can we use neural prostheses to stimulate specific muscles in order to help recovery of motor control after stroke or cervical injury? This question is of crucial importance to 15 million people each year around the globe, and is at the heart of Numa Dancause and Marco Bonizzato’s research, our collaborators in the Neuroscience department at the University of Montreal. It is now possible to implant large capacity electrodes for electrical stimulation in cortex, but still difficult to predict their effect on the brain and the rest of the body. Nevertheless, preliminary but promising results on rats and monkeys have shown that a non-negligible motor recovery is obtained after stimulation of regions of motor cortex that are still functional. The difficulties related to optimal microcortical stimulation hence consist in finding both one of these regions, and a stimulation protocol with optimal recovery efficacy. This search has up to present day been performed by hand, but recent and upcoming large scale stimulation technologies permitting delivery of spatio-temporal signals are making such exhaustive searches impossible.A promising approach to automating and optimizing this discovery is the use of Bayesian optimization. My work has consisted in developing and refining such techniques with two scientific questions in mind: (1) how can we evoke complex movements by chaining cortical microstimulations?, and (2) can these outperform single channel stimulations in terms of recovery efficacy? We present in the main article of this thesis our hierarchical Bayesian optimization approach which uses gaussian processes to exploit known properties of the brain to speed up the search, as well as first results answering question 1. We leave to future work a definitive answer to the second question.

Machine learning for image classification is a hot topic and it is increasing in popularity. Therefore the aim of this study is to provide a better understanding of convolutional neural network hyperparameters by comparing the test accuracy of convolutional neural network models with different hyperparameter value configurations. The focus of this study is to see whether there is an influence in the learning process depending on which hyperparameter values were used. For conducting the experiments convolutional neural network models were developed using the programming language Python utilizing the library Keras. The dataset used for this study iscifar-10, it includes 60000 colour images of 10 categories ranging from man-made objects to different animal species. Grid search is used for instantiating models with varying learning rate and momentum, width and depth values. Learning rate is only tested combined with momentum and width is only tested combined with depth. Activation functions, convolutional layers and batch size are tested individually. Grid search is compared against Bayesian optimization to see which technique will find the most optimized learning rate and momentum values. Results illustrate that the impact different hyperparameters have on the overall test accuracy varies. Learning rate and momentum affects the test accuracy greatly, however suboptimal values for learning rate and momentum can decrease the test accuracy severely. Activation function, width and depth, convolutional layer and batch size have a lesser impact on test accuracy. Regarding Bayesian optimization compared to grid search, results show that Bayesian optimization will not necessarily find more optimal hyperparameter values.

<p>Practical design problems in engineering and science involve the evaluation of expensive black-box functions, the optimization of multiple—often conflicting—targets, and the integration of data generated by multiple sources of information, e.g., numerical models with different levels of fidelity. If not properly handled, the complexity of these design problems can lead to lengthy and costly development cycles. In the last years, Bayesian optimization has emerged as a powerful alternative to solve optimization problems that involve the evaluation of expensive black-box functions. Bayesian optimization has two main components: a probabilistic surrogate model of the black-box function and an acquisition function that drives the optimization. Its ability to find high-performance designs within a limited number of function evaluations has attracted the attention of many fields including the engineering design community. The practical relevance of strategies with the ability to fuse information emerging from different sources and the need to optimize multiple targets has motivated the development of multi-fidelity modeling techniques and multi-objective Bayesian optimization methods. A key component in the vast majority of these methods is the Gaussian process (GP) due to its flexibility and mathematical properties.</p> <p><br></p> <p>The objective of this dissertation is to develop new approaches in the areas of multi-fidelity modeling and multi-objective Bayesian optimization. To achieve this goal, this study explores the use of linear and non-linear compositions of GPs to build probabilistic models for Bayesian optimization. Additionally, motivated by the rationale behind well-established multi-objective methods, this study presents a novel acquisition function to solve multi-objective optimization problems in a Bayesian framework. This dissertation presents four contributions. First, the auto-regressive model, one of the most prominent multi-fidelity models in engineering design, is extended to include informative mean functions that capture prior knowledge about the global trend of the sources. This additional information enhances the predictive capabilities of the surrogate. Second, the non-linear auto-regressive Gaussian process (NARGP) model, a non-linear multi-fidelity model, is integrated into a multi-objective Bayesian optimization framework. The NARGP model offers the possibility to leverage sources that present non-linear cross-correlations to enhance the performance of the optimization process. Third, GP classifiers, which employ non-linear compositions of GPs, and conditional probabilities are combined to solve multi-objective problems. Finally, a new multi-objective acquisition function is presented. This function employs two terms: a distance-based metric—the expected Pareto distance change—that captures the optimality of a given design, and a diversity index that prevents the evaluation of non-informative designs. The proposed acquisition function generates informative landscapes that produce Pareto front approximations that are both broad and diverse.</p>

[pt] Os risers são importantes componentes na produção e exploração de petróleo e derivados. São responsáveis pelo transporte do óleo e gás encontrados no reservatório até a Unidade Estacionária de Produção (UEP) ou pela injeção de gás ou água no reservatório. A crescente demanda por esse produto faz com que a exploração seja feita em regiões com condições cada vez mais adversas. Tipicamente, um projeto deste porte exige um número muito grande de análises numéricas de elementos finitos e exigem uma experiência grande por parte do projetista a fim de obter uma solução viável. Esse desafio leva engenheiros a buscarem ferramentas consistentes e seguras que auxiliem nas etapas iniciais do projeto das configurações de risers e que sejam capazes de diminuir o número de análises totais exigidas. Uma dessas ferramentas é a utilização de métodos de otimização para obter de maneira consistente e segura os parâmetros que definem uma configuração. Este trabalho apresenta o método de Otimização Bayesiana, um método baseado em técnicas de aprendizado de máquina capaz de resolver problemas de otimização do tipo caixa-preta de maneira eficiente explorando o uso de aproximações analíticas da função objetivo, que se quer otimizar. O método é aplicado em diferentes estudos de casos visando validálo como capaz de resolver problemas de configuração de riser de maneira eficiente e consistente. Dentre os problemas aplicados estão diferentes tipos de configurações, diferentes casos realistas, mono-objetivo e multi-objetivo. / [en] Risers are an important component in the oil s production and exploration field. They are responsible for the oil and gas transportation from the reservoir to the floating unit or injection of gas or water into the reservoir. The increasing the demand for this product has lead projects to explore to areas in which conditions are harsher. Typically, such a large project demands a large number of numerical finite element analyses and a great expertise from the engineer in charge in order to obtain a viable solution. This challenge leads engineers in search of consistent and reliable tools that assist in the early stages of the riser configuration design and are capable of reducing the number of total analyses required. One of these tools is application of optimization methods to obtain in a consistent and reliable manner the parameters which define a configuration. This work presents the Bayesian Optimization method, a method based on machine learning techniques capable of efficiently solving so called black box problems by exploring analytical approximations of the objective function, the function to be minimized. The method is applied to different case studies aiming to validate it as capable of solving a wide variety of riser configuration problems in an efficient and consistent way. Among the problems applied are different types of configurations, different realistic cases, mono-objective and multi-objective.

L’apprentissage par renforcement profond a mené à de nombreux résultats prometteurs dans l’apprentissage des jeux vidéo à partir de pixels, dans la robotique pour l’apprentissage de compétences généralisables et dans les soins de santé pour l’apprentissage de traitement dynamiques. Un obstacle demeure toutefois: celui du manque d’efficacité dans le nombre d’échantillons nécessaires pour obtenir de bons résultats. Pour résoudre ce problème, notre objectif est d’améliorer l’efficacité de l’apprentissage en améliorant les capacité d’acquisition de nouvelles données, un problème d’exploration. L’approche proposée consiste à : (1) Apprendre un ensemble diversifié d’environments (donnant lieu à un changement de dynamique) (2) Apprendre une politique capable de mieux s’adapter aux changements dans l’envi- ronnement, à l’aide du méta-apprentissage. Cette méthode peut avoir des impacts bénéfiques dans de nombreux problèmes du monde réel tels que la découverte de médicaments, dans laquelle nous sommes confrontés à un espace d’actions très grand. D’autant plus, la conception de nouvelles substances thérapeutiques qui sont fonctionnellement intéressantes nécessite une exploration efficace du paysage de la recherche. / Deep reinforcement learning has led to promising results in learning video games from pixels, robotics for learning generalizable skills, and healthcare for learning dynamic treatments. However, an obstacle remains the lack of efficiency in the number of samples required to achieve good results. To address this problem, our goal is to improve sample efficiency by improving the ability to acquire new data, an issue of exploration. The proposed approach is to: (1) Learn a diverse set of environments (resulting in a change of dynamics) (2) earn a policy that can better adapt to changes in the environment using meta-learning This method can benefit many real-world problems, such as drug discovery, where we face a large action space. Furthermore, designing new therapeutic substances that are functionally interesting requires efficient exploration of the research landscape

In the fields of process control engineering and robotics, especially in automatic control, optimization challenges frequently manifest as complex problems with expensive evaluations. This thesis zeroes in on one such problem: the optimization of camera positions for Convolutional Neural Networks (CNNs). CNNs have specific attention points in images that are often not intuitive to human perception, making camera placement critical for performance. The research is guided by two primary questions. The first investigates the role of Explainable Artificial Intelligence (XAI), specifically GradCAM++ visual explanations, in Computer Vision for aiding in the evaluation of different camera positions. Building on this, the second question assesses a novel algorithm that leverages these XAI features against traditional black-box optimization methods. To answer these questions, the study employs a robotic auto-positioning system for data collection, CNN model training, and performance evaluation. A case study focused on classifying flow regimes in industrial-grade bioreactors validates the method. The proposed approach shows improvements over established techniques like Grid Search, Random Search, Bayesian optimization, and Simulated Annealing. Future work will focus on gathering more data and including noise for generalized conclusions.:Contents 1 Introduction 1.1 Motivation 1.2 Problem Analysis 1.3 Research Question 1.4 Structure of the Thesis 2 State of the Art 2.1 Literature Research Methodology 2.1.1 Search Strategy 2.1.2 Inclusion and Exclusion Criteria 2.2 Blackbox Optimization 2.3 Mathematical Notation 2.4 Bayesian Optimization 2.5 Simulated Annealing 2.6 Random Search 2.7 Gridsearch 2.8 Explainable A.I. and Saliency Maps 2.9 Flowregime Classification in Stirred Vessels 2.10 Performance Metrics 2.10.1 R2 Score and Polynomial Regression for Experiment Data Analysis 2.10.2 Blackbox Optimization Performance Metrics 2.10.3 CNN Performance Metrics 3 Methodology 3.1 Requirement Analysis and Research Hypothesis 3.2 Research Approach: Case Study 3.3 Data Collection 3.4 Evaluation and Justification 4 Concept 4.1 System Overview 4.2 Data Flow 4.3 Experimental Setup 4.4 Optimization Challenges and Approaches 5 Data Collection and Experimental Setup 5.1 Hardware Components 5.2 Data Recording and Design of Experiments 5.3 Data Collection 5.4 Post-Experiment 6 Implementation 6.1 Simulation Unit 6.2 Recommendation Scalar from Saliency Maps 6.3 Saliency Map Features as Guidance Mechanism 6.4 GradCam++ Enhanced Bayesian Optimization 6.5 Benchmarking Unit 6.6 Benchmarking 7 Results and Evaluation 7.1 Experiment Data Analysis 7.2 Recommendation Scalar 7.3 Benchmarking Results and Quantitative Analysis 7.3.1 Accuracy Results from the Benchmarking Process 7.3.2 Cumulative Results Interpretation 7.3.3 Analysis of Variability 7.4 Answering the Research Questions 7.5 Summary 8 Discussion 8.1 Critical Examination of Limitations 8.2 Discussion of Solutions to Limitations 8.3 Practice-Oriented Discussion of Findings 9 Summary and Outlook / Im Bereich der Prozessleittechnik und Robotik, speziell bei der automatischen Steuerung, treten oft komplexe Optimierungsprobleme auf. Diese Arbeit konzentriert sich auf die Optimierung der Kameraplatzierung in Anwendungen, die Convolutional Neural Networks (CNNs) verwenden. Da CNNs spezifische, für den Menschen nicht immer ersichtliche, Merkmale in Bildern hervorheben, ist die intuitive Platzierung der Kamera oft nicht optimal. Zwei Forschungsfragen leiten diese Arbeit: Die erste Frage untersucht die Rolle von Erklärbarer Künstlicher Intelligenz (XAI) in der Computer Vision zur Bereitstellung von Merkmalen für die Bewertung von Kamerapositionen. Die zweite Frage vergleicht einen darauf basierenden Algorithmus mit anderen Blackbox-Optimierungstechniken. Ein robotisches Auto-Positionierungssystem wird zur Datenerfassung und für Experimente eingesetzt. Als Lösungsansatz wird eine Methode vorgestellt, die XAI-Merkmale, insbesondere solche aus GradCAM++ Erkenntnissen, mit einem Bayesschen Optimierungsalgorithmus kombiniert. Diese Methode wird in einer Fallstudie zur Klassifizierung von Strömungsregimen in industriellen Bioreaktoren angewendet und zeigt eine gesteigerte performance im Vergleich zu etablierten Methoden. Zukünftige Forschung wird sich auf die Sammlung weiterer Daten, die Inklusion von verrauschten Daten und die Konsultation von Experten für eine kostengünstigere Implementierung konzentrieren.:Contents 1 Introduction 1.1 Motivation 1.2 Problem Analysis 1.3 Research Question 1.4 Structure of the Thesis 2 State of the Art 2.1 Literature Research Methodology 2.1.1 Search Strategy 2.1.2 Inclusion and Exclusion Criteria 2.2 Blackbox Optimization 2.3 Mathematical Notation 2.4 Bayesian Optimization 2.5 Simulated Annealing 2.6 Random Search 2.7 Gridsearch 2.8 Explainable A.I. and Saliency Maps 2.9 Flowregime Classification in Stirred Vessels 2.10 Performance Metrics 2.10.1 R2 Score and Polynomial Regression for Experiment Data Analysis 2.10.2 Blackbox Optimization Performance Metrics 2.10.3 CNN Performance Metrics 3 Methodology 3.1 Requirement Analysis and Research Hypothesis 3.2 Research Approach: Case Study 3.3 Data Collection 3.4 Evaluation and Justification 4 Concept 4.1 System Overview 4.2 Data Flow 4.3 Experimental Setup 4.4 Optimization Challenges and Approaches 5 Data Collection and Experimental Setup 5.1 Hardware Components 5.2 Data Recording and Design of Experiments 5.3 Data Collection 5.4 Post-Experiment 6 Implementation 6.1 Simulation Unit 6.2 Recommendation Scalar from Saliency Maps 6.3 Saliency Map Features as Guidance Mechanism 6.4 GradCam++ Enhanced Bayesian Optimization 6.5 Benchmarking Unit 6.6 Benchmarking 7 Results and Evaluation 7.1 Experiment Data Analysis 7.2 Recommendation Scalar 7.3 Benchmarking Results and Quantitative Analysis 7.3.1 Accuracy Results from the Benchmarking Process 7.3.2 Cumulative Results Interpretation 7.3.3 Analysis of Variability 7.4 Answering the Research Questions 7.5 Summary 8 Discussion 8.1 Critical Examination of Limitations 8.2 Discussion of Solutions to Limitations 8.3 Practice-Oriented Discussion of Findings 9 Summary and Outlook

No description available.

<p dir="ltr">Broadly, the focus of this work is on efficient statistical estimation and optimization of data arising from experimental data, particularly motivated by nanomanufacturing experiments on the material tellurene. Tellurene is a novel material for transistors with reliable attributes that enhance the performance of electronics (e.g., nanochip). As a solution-grown product, two-dimensional (2D) tellurene can be manufactured through a scalable process at a low cost. There are three main throughlines to this work, data augmentation, optimization, and equality constraint, and three distinct methodological projects, each of which addresses a subset of these throughlines. For the first project, I apply transfer learning in the analysis of data from a new tellurene experiment (process B) using the established linear regression model from a prior experiment (process A) from a similar study to combine the information from both experiments. The key of this approach is to incorporate the total equivalent amounts (TEA) of a lurking variable (experimental process changes) in terms of an observed (base) factor that appears in both experimental designs into the prespecified linear regression model. The results of the experimental data are presented including the optimal PVP chain length for scaling up production through a larger autoclave size. For the second project, I develop a multi-armed bandit Bayesian optimization (BO) approach to incorporate the equality constraint that comes from a mixture experiment on tellurium nanoproduct and account for factors with categorical levels. A more complex optimization approach was necessitated by the experimenters’ use of a neural network regression model to estimate the response surface. Results are presented on synthetic data to validate the ability of BO to recover the optimal response and its efficiency is compared to Monte Carlo random sampling to understand the level of experimental design complexity at which BO begins to pay off. The third project examines the potential enhancement of parameter estimation by utilizing synthetic data generated through Generative Adversarial Networks (GANs) to augment experimental data coming from a mixture experiment with a small to moderate number of runs. Transfer learning shows high promise for aiding in tellurene experiments, BO’s value increases with the complexity of the experiment, and GANs performed poorly on smaller experiments introducing bias to parameter estimates.</p>

Notwithstanding the recent successes of deep convolutional neural networks for classification tasks, they are sensitive to the selection of their hyperparameters, which impose an exponentially large search space on modern convolutional models. Traditional hyperparameter selection methods include manual, grid, or random search, but these require expert knowledge or are computationally burdensome. Divergently, Bayesian optimization and evolutionary inspired techniques have surfaced as viable alternatives to the hyperparameter problem. Thus, an alternative hybrid approach that combines the advantages of these techniques is proposed. Specifically, the search space is partitioned into discrete-architectural, and continuous and categorical hyperparameter subspaces, which are respectively traversed by a stochastic genetic search, followed by a genetic-Bayesian search. Simulations on a prominent image classification task reveal that the proposed method results in an overall classification accuracy improvement of 0.87% over unoptimized baselines, and a greater than 97% reduction in computational costs compared to a commonly employed brute force approach. / Electrical and Mining Engineering / M. Tech. (Electrical Engineering)