This thesis introduces the concept of Bayesian optimization, primarly used in optimizing costly black-box functions. Besides theoretical treatment of the topic, the focus of the thesis is on two numerical experiments. Firstly, different types of acquisition functions, which are the key components responsible for the performance, are tested and compared. Special emphasis is on the analysis of a so-called exploration-exploitation trade-off. Secondly, one of the most recent applications of Bayesian optimization concerns hyperparameter tuning in machine learning algorithms, where the objective function is expensive to evaluate and not given analytically. However, some results indicate that much simpler methods can give similar results. Our contribution is therefore a statistical comparison of simple random search and Bayesian optimization in the context of finding the optimal set of hyperparameters in support vector regression. It has been found that there is no significant difference in performance of these two methods.

Machine learning solutions have been successfully used to solve many simple and complex problems. However, their development process still relies on human experts to perform tasks such as data preprocessing, feature engineering and model selection. As the complexity of these tasks increases, so does the demand for automated solutions, namely Automated Machine Learning (AutoML). Most algorithms employed in these systems have hyperparameters whose configuration may directly affect their predictive performance. Therefore, hyperparameter tuning is a recurring task in AutoML systems. This thesis investigated how to efficiently automate hyperparameter tuning by means of Meta-learning. To this end, large-scale experiments were performed tuning the hyperparameters of different classification algorithms, and an enhanced experimental methodology was adopted throughout the thesis to explore and learn the hyperparameter profiles for different classification algorithms. The results also showed that in many cases the default hyperparameter settings induced models that are on par with those obtained by tuning. Hence, a new Meta-learning recommender system was proposed to identify when it is better to use default values and when to tune classification algorithms for each new dataset. The proposed system is capable of generalizing several learning processes into a single modular framework, along with the possibility of assigning different algorithms. Furthermore, a descriptive analysis of model predictions is used to identify which data characteristics affect the necessity for tuning in each one of the algorithms investigated in the thesis. Experimental results also demonstrated that the proposed recommender system reduced the time spent on optimization processes, without reducing the predictive performance of the induced models. Depending on the target algorithm, the Meta-learning recommender system can statistically outperform the baselines. The significance of these results opens a number of new avenues for future work. / Soluções de aprendizado de máquina tem sido cada vez mais usadas com sucesso para resolver problemas dos mais simples aos complexos. Entretanto, o processo de desenvolvimento de tais soluções ainda é um processo que depende da ação de especialistas humanos em tarefas como: pré-processamento dos dados, engenharia de features e seleção de modelos. Consequentemente, quando a complexidade destas tarefas atinge um nível muito alto, há a necessidade de soluções automatizadas, denominadas por Aprendizado de Máquina automatizado (AutoML). A maioria dos algoritmos usados em tais sistemas possuem hiper-parâmetros cujos valores podem afetar diretamente o desempenho preditivo dos modelos gerados. Assim sendo, o ajuste de hiper-parâmetros é uma tarefa recorrente no desenvolvimento de sistems de AutoML. Nesta tese investigou-se a automatização do ajuste de hiper-parâmetros por meio de Meta-aprendizado. Seguindo essa linha, experimentos massivos foram realizados para ajustar os hiper-parâmetros de diferentes algoritmos de classificação. Além disso, uma metodologia experimental aprimorada e adotada ao lngo da tese perimtiu identificar diferentes perfis de ajuste para diferentes algoritmos de classificação. Entretanto, os resultados também mostraram que em muitos casos as configurações default destes algoritmos induziram modelos mais precisos do que os obtidos por meio de ajuste. Assim, foi proposto um novo sistema de recomendação baseado em Meta-learning para identificar quando é melhor realizar o ajuste de parâmetros para os algoritmos de classificação ou apenas usar os valores default. O sistema proposto é capaz de generalizar várias etapas do aprendizado em um único framework modular, juntamente com a possibilidade de avaliar diferentes algoritmos de aprendizado de máquina. As análises descritivas das predições obtidas pelo sistema indicaram quais características podem ser responsáveis por determinar quando o ajuste se faz necessário para cada um dos algoritmos investigados na tese. Os resultados também demonstraram que o sistema recomendador proposto reduziu o tempo gasto com a otimização mantendo o desempenho preditivo dos modelos gerados. Além disso, dependendo do algoritmo de classificação modelado, o sistema foi estatisticamente superior aos baselines. A significância desdes resultados abre um novo número de oportunidades para trabalhos futuros.

Machine learning solutions have been successfully used to solve many simple and complex problems. However, their development process still relies on human experts to perform tasks such as data preprocessing, feature engineering and model selection. As the complexity of these tasks increases, so does the demand for automated solutions, namely Automated Machine Learning (AutoML). Most algorithms employed in these systems have hyperparameters whose configuration may directly affect their predictive performance. Therefore, hyperparameter tuning is a recurring task in AutoML systems. This thesis investigated how to efficiently automate hyperparameter tuning by means of Meta-learning. To this end, large-scale experiments were performed tuning the hyperparameters of different classification algorithms, and an enhanced experimental methodology was adopted throughout the thesis to explore and learn the hyperparameter profiles for different classification algorithms. The results also showed that in many cases the default hyperparameter settings induced models that are on par with those obtained by tuning. Hence, a new Meta-learning recommender system was proposed to identify when it is better to use default values and when to tune classification algorithms for each new dataset. The proposed system is capable of generalizing several learning processes into a single modular framework, along with the possibility of assigning different algorithms. Furthermore, a descriptive analysis of model predictions is used to identify which data characteristics affect the necessity for tuning in each one of the algorithms investigated in the thesis. Experimental results also demonstrated that the proposed recommender system reduced the time spent on optimization processes, without reducing the predictive performance of the induced models. Depending on the target algorithm, the Meta-learning recommender system can statistically outperform the baselines. The significance of these results opens a number of new avenues for future work. / Soluções de aprendizado de máquina tem sido cada vez mais usadas com sucesso para resolver problemas dos mais simples aos complexos. Entretanto, o processo de desenvolvimento de tais soluções ainda é um processo que depende da ação de especialistas humanos em tarefas como: pré-processamento dos dados, engenharia de features e seleção de modelos. Consequentemente, quando a complexidade destas tarefas atinge um nível muito alto, há a necessidade de soluções automatizadas, denominadas por Aprendizado de Máquina automatizado (AutoML). A maioria dos algoritmos usados em tais sistemas possuem hiper-parâmetros cujos valores podem afetar diretamente o desempenho preditivo dos modelos gerados. Assim sendo, o ajuste de hiper-parâmetros é uma tarefa recorrente no desenvolvimento de sistems de AutoML. Nesta tese investigou-se a automatização do ajuste de hiper-parâmetros por meio de Meta-aprendizado. Seguindo essa linha, experimentos massivos foram realizados para ajustar os hiper-parâmetros de diferentes algoritmos de classificação. Além disso, uma metodologia experimental aprimorada e adotada ao lngo da tese perimtiu identificar diferentes perfis de ajuste para diferentes algoritmos de classificação. Entretanto, os resultados também mostraram que em muitos casos as configurações default destes algoritmos induziram modelos mais precisos do que os obtidos por meio de ajuste. Assim, foi proposto um novo sistema de recomendação baseado em Meta-learning para identificar quando é melhor realizar o ajuste de parâmetros para os algoritmos de classificação ou apenas usar os valores default. O sistema proposto é capaz de generalizar várias etapas do aprendizado em um único framework modular, juntamente com a possibilidade de avaliar diferentes algoritmos de aprendizado de máquina. As análises descritivas das predições obtidas pelo sistema indicaram quais características podem ser responsáveis por determinar quando o ajuste se faz necessário para cada um dos algoritmos investigados na tese. Os resultados também demonstraram que o sistema recomendador proposto reduziu o tempo gasto com a otimização mantendo o desempenho preditivo dos modelos gerados. Além disso, dependendo do algoritmo de classificação modelado, o sistema foi estatisticamente superior aos baselines. A significância desdes resultados abre um novo número de oportunidades para trabalhos futuros.

No description available.

Using the data collected during the hyperparameter tuning for Google's Dependency Parser Parsey McParseface, Feedforward neural networks and the correlation between its hyperparameter during the networks training are explained and analysed in depth.:1 Introduction to Neural Networks 4 1.1 History of AI 4 1.2 The role of Neural Networks in AI Research 6 1.2.1 Artificial Intelligence 6 1.2.2 Machine Learning 6 1.2.3 Neural Network 8 1.3 Structure of Neural Networks 8 1.3.1 Biology Analogy of Artificial Neural Networks 9 1.3.2 Architecture of Artificial Neural Networks 9 1.3.3 Biological Model of Nodes – Neurons 11 1.3.4 Structure of Artificial Neurons 12 1.4 Training a Neural Network 21 1.4.1 Data 21 1.4.2 Hyperparameters 22 1.4.3 Training process 26 1.4.4 Overfitting 27 2 Natural Language Processing (NLP) 29 2.1 Data Preparation 29 2.1.1 Text Preprocessing 29 2.1.2 Part-of-Speech Tagging 30 2.2 Dependency Parsing 31 2.2.1 Dependency Grammar 31 2.2.2 Dependency Parsing Rule-Based & Data-Driven Approach 33 2.2.3 Syntactic Parser 33 2.3 Parsey McParseface 34 2.3.1 SyntaxNet 34 2.3.2 Corpus 34 2.3.3 Architecture 34 2.3.4 Improvements to the Feed Forward Neural Network 38 3 Training of Parsey’s Cousins 41 3.1 Training a Model 41 3.1.1 Building the Framework 41 3.1.2 Corpus 41 3.1.3 Training Process 43 3.1.4 Settings for the Training 44 3.2 Results and Analysis 46 3.2.1 Results from Google’s Models 46 3.2.2 Effect of Hyperparameter 47 4 Conclusion 63 5 Bibliography 65 6 Appendix 74

This thesis develops a method for using machine learning in a industrial pro-cess. The implementation of this machine learning model aimed to reduce costsand increase efficiency of train wheel overhaul in partnership with the AustrianFederal Railroads, Oebb. Different machine learning models as well as categoryencodings were tested to find which performed best on the data set. In addition,differently sized training sets were used to determine whether size of the trainingset affected the results. The implementation shows that Oebb can save moneyand increase efficiency of train wheel overhaul by using machine learning andthat continuous training of prediction models is necessary because of variationsin the data set.

Side-channel attacks have become a realistic threat to the implementations of cryptographic algorithms. By analyzing the unintentional, side-channel leakage, the attacker is able to recover the secret of the target. Recently, a new type of side-channel leakage has been discovered, called far field EM emissions. Unlike attacks based on near field EM emissions or power consumption, the attack based on far field EM emissions is able to extract the secret key from the victim device of several meters distance. However, existing deep-learning attacks based far field EM commonly use a random or grid search method to optimize neural networks’ hyperparameters. Recently, an automated way for deep learning hyperparameter tuning based on Auto- Keras library, called AutoSCA framework, was applied to near-field EM attacks. In this work, we investigate if AutoSCA could help far field EM side-channel attacks. In our experiments, the target is a Bluetooth-5 supported Nordic Semiconductor nRF52832 development kit implementation of Advanced Encryption Standard (AES). Our experiments show that, by using a deep-learning model generated by the AutoSCA framework, we need 485 traces on average to recover a subkey from traces captured at 15 meters distance from the victim device without repeating each encryption. For the same conditions, the state-of-the-art method uses 510 traces. Furthermore, our model contains only 667,433 trainable parameters in total, implying that it requires roughly 9 times less training resources compared to the larger models in the previous work. / Angrepp på sidokanaler har blivit ett realistiskt hot mot implementeringen av kryptografiska algoritmer.Genom att analysera det oavsiktliga läckaget kan angriparen hitta hemligheten bakom målet.Nyligen har en ny typ av sidokanalläckage upptäckts, kallad fjärrfälts EM-utsläpp.Till skillnad från attacker baserade på nära fält EM- utsläpp eller energiförbrukning, kan attacken baserad på yttre fält EM-utsläpp extrahera den hemliga nyckeln från den skadade anordningen på flera meter avstånd.Men befintliga djupinlärningsattacker baserade på långt fält EM använder ofta en slumpmässig sökmetod för att optimera nervnätens hyperparametrar. Nyligen tillämpades ett automatiserat sätt för djupinlärning av hyperparametern baserad på Auto-Keras- bibliotek, AutoSCA- ramverket, vid EM-angrepp nära fältet.I det här arbetet undersöker vi om AutoSCA kan hjälpa till med EM-angrepp.I våra experiment är målet en Bluetooth-5-stödd nordisk semidirigent nR52832- utvecklingsutrustning för avancerad krypteringsstandard (AES).Våra experiment visar att genom att använda en djupinlärningsmodell skapad av AutoSCA-ramverket, behöver vi 485-spår i genomsnitt för att hämta en subnyckel från spår tagna på 15- meters avstånd från offrets apparat utan att upprepa varje kryptering.Under samma förhållanden använder den senaste metoden 510-spår.Dessutom innehåller vår modell bara 667,433-parametrar som totalt kan användas för utbildning, vilket innebär att det krävs ungefär nio gånger mindre utbildningsresurser jämfört med de större modellerna i det tidigare arbetet.

No description available.

Assessing the financial health of organizations remains a topic of great interest to economists, financial institutions, and invested stakeholders. For more than a century, research into financial distress has focused primarily on traditional applications of statistical analysis; however, modern advances in computational efficiency have created a significant opportunity for more sophisticated approaches. This thesis investigates the application of artificial intelligence on company bankruptcy prediction. The proposed neural network model is evaluated using the Polish Companies Bankruptcy dataset and yields a 5-year prediction accuracy of 96.5% and an AUC (area under receiver operating characteristic curve) measure of 92.4%.

Les algorithmes d'inférence ou d'optimisation possèdent généralement des hyperparamètres qu'il est nécessaire d'ajuster. Nous nous intéressons ici à l'automatisation de cette étape d'ajustement et considérons différentes méthodes qui y parviennent en apprenant en ligne la structure du problème considéré.La première moitié de cette thèse explore l'ajustement des hyperparamètres en apprentissage artificiel. Après avoir présenté et amélioré le cadre générique de l'optimisation séquentielle à base de modèles (SMBO), nous montrons que SMBO s'applique avec succès à l'ajustement des hyperparamètres de réseaux de neurones profonds. Nous proposons ensuite un algorithme collaboratif d'ajustement qui mime la mémoire qu'ont les humains d'expériences passées avec le même algorithme sur d'autres données.La seconde moitié de cette thèse porte sur les algorithmes MCMC adaptatifs, des algorithmes d'échantillonnage qui explorent des distributions de probabilité souvent complexes en ajustant leurs paramètres internes en ligne. Pour motiver leur étude, nous décrivons d'abord l'observatoire Pierre Auger, une expérience de physique des particules dédiée à l'étude des rayons cosmiques. Nous proposons une première partie du modèle génératif d'Auger et introduisons une procédure d'inférence des paramètres individuels de chaque événement d'Auger qui ne requiert que ce premier modèle. Ensuite, nous remarquons que ce modèle est sujet à un problème connu sous le nom de label switching. Après avoir présenté les solutions existantes, nous proposons AMOR, le premier algorithme MCMC adaptatif doté d'un réétiquetage en ligne qui résout le label switching. Nous présentons une étude empirique et des résultats théoriques de consistance d'AMOR, qui mettent en lumière des liens entre le réétiquetage et la quantification vectorielle / Inference and optimization algorithms usually have hyperparameters that require to be tuned in order to achieve efficiency. We consider here different approaches to efficiently automatize the hyperparameter tuning step by learning online the structure of the addressed problem. The first half of this thesis is devoted to hyperparameter tuning in machine learning. After presenting and improving the generic sequential model-based optimization (SMBO) framework, we show that SMBO successfully applies to the task of tuning the numerous hyperparameters of deep belief networks. We then propose an algorithm that performs tuning across datasets, mimicking the memory that humans have of past experiments with the same algorithm on different datasets. The second half of this thesis deals with adaptive Markov chain Monte Carlo (MCMC) algorithms, sampling-based algorithms that explore complex probability distributions while self-tuning their internal parameters on the fly. We start by describing the Pierre Auger observatory, a large-scale particle physics experiment dedicated to the observation of atmospheric showers triggered by cosmic rays. The models involved in the analysis of Auger data motivated our study of adaptive MCMC. We derive the first part of the Auger generative model and introduce a procedure to perform inference on shower parameters that requires only this bottom part. Our model inherently suffers from label switching, a common difficulty in MCMC inference, which makes marginal inference useless because of redundant modes of the target distribution. After reviewing existing solutions to label switching, we propose AMOR, the first adaptive MCMC algorithm with online relabeling. We give both an empirical and theoretical study of AMOR, unveiling interesting links between relabeling algorithms and vector quantization.