Validation and verification in machine learning is an open problem which becomes increasingly important as its applications becomes more critical. Amongst the applications are autonomous vehicles and medical diagnostics. These systems all needs to be validated before being put into use or else the consequences might be fatal. This master’s thesis focuses on improving both learning and validating machine learning models in cases where data can either be generated or collected based on a chosen position. This can for example be taking and labeling photos at the position or running some simulation which generates data from the chosen positions. The approach is twofold. The first part concerns modeling the relationship between any fixed-size set of positions and some real valued performance measure. The second part involves calculating such a performance measure by estimating the performance over a region of positions. The result is two different algorithms, both variations of Bayesian optimization. The first algorithm models the relationship between a set of points and some performance measure while also optimizing the function and thus finding the set of points which yields the highest performance. The second algorithm uses Bayesian optimization to approximate the integral of performance over the region of interest. The resulting algorithms are validated in two different simulated environments. The resulting algorithms are applicable not only to machine learning but can also be used to optimize any function which takes a set of positions and returns a value, but are more suitable when the function is expensive to evaluate.

In this work we present a novel approach to network embedding for neural architecture search - info-NAS. The model learns to predict the output fea- tures of a trained convolutional neural network on a set of input images. We use the NAS-Bench-101 search space as the neural architecture dataset, and the CIFAR-10 as the image dataset. For the purpose of this task, we extend an existing unsupervised graph variational autoencoder, arch2vec, by jointly training on unlabeled and labeled neural architectures in a semi-supervised manner. To evaluate our approach, we analyze how our model learns on the data, compare it to the original arch2vec, and finally, we evaluate both mod- els on the NAS-Bench-101 search task and on the performance prediction task. 1

Machine Learning has become more important for materials engineering in the last decade. Globally, automated machine learning (AutoML) is growing in popularity with the increasing demand for data analysis solutions. Yet, it is not frequently used for small tabular data. Comparisons and benchmarks already exist to assess the qualities of AutoML tools in general, but none of them elaborates on the surrounding conditions of materials engineers working with experimental data: small datasets with less than 1000 samples. This benchmark addresses these conditions and draws special attention to the overall competitiveness with manual data analysis. Four representative AutoML frameworks are used to evaluate twelve domain-specific datasets to provide orientation on the promises of AutoML in the field of materials engineering. Performance, robustness and usability are discussed in particular. The results lead to two main conclusions: First, AutoML is highly competitive with manual model optimization, even with little training time. Second, the data sampling for train and test data is of crucial importance for reliable results.

El aprendizaje de máquinas ha ganado terreno utilizándose en casi todas las áreas de la vida cotidiana, ayudando a tomar decisiones en las finanzas, la medicina, el comercio y el entretenimiento. El desarrollo continuo de nuevos algoritmos y técnicas de aprendizaje automático, y la amplia gama de herramientas y conjuntos de datos disponibles han traído nuevas oportunidades y desafíos para investigadores y profesionales tanto del mundo académico como de la industria. Seleccionar la mejor estrategia posible para resolver un problema de aprendizaje automático es cada vez más difícil, en parte porque requiere largos tiempos de experimentación y profundos conocimientos técnicos. En este escenario, el campo de investigación Automated Machine Learning (AutoML) ha ganado protagonismo, proponiendo estrategias para automatizar progresivamente tareas usuales durante el desarrollo de aplicaciones de aprendizaje de máquina. Las herramientas de AutoML más comunes permiten seleccionar automáticamente dentro de un conjunto restringido de algoritmos y parámetros la mejor estrategia para cierto conjunto de datos. Sin embargo, los problemas prácticos a menudo requieren combinar y comparar algoritmos heterogéneos implementados con diferentes tecnologías subyacentes. Un ejemplo es el procesamiento del lenguaje natural, un escenario donde varía el espacio de posibles técnicas a aplicar ampliamente entre diferentes tareas, desde el preprocesamiento hasta la representación y clasificación de textos. Realizar AutoML en un escenario heterogéneo como este es complejo porque la solución necesaria podría incluir herramientas y bibliotecas no compatibles entre sí. Esto requeriría que todos los algoritmos acuerden un protocolo común que permita la salida de un algoritmo para ser compartida como entradas a cualquier otro. En esta investigación se diseña e implementa un sistema de AutoML que utiliza técnicas heterogéneas. A diferencia de los enfoques de AutoML existentes, nuestra contribución puede combinar técnicas y algoritmos de diferentes bibliotecas y tecnologías, incluidos algoritmos de aprendizaje de máquina clásicos, extracción de características, herramientas de procesamiento de lenguaje natural y diversas arquitecturas de redes neuronales. Definimos el problema heterogéneo de optimización de AutoML como la búsqueda de la mejor secuencia de algoritmos que transforme datos de entrada específicos en la salida deseada. Esto proporciona un enfoque teórico y práctico novedoso para AutoML. Nuestra propuesta se evalúa experimentalmente en diversos problemas de aprendizaje automático y se compara con enfoques alternativos, lo que demuestra que es competitiva con otras alternativas de AutoML en los puntos de referencia estándar. Además, se puede aplicar a escenarios novedosos, como varias tareas de procesamiento de lenguaje natural, donde las alternativas existentes no se pueden implementar directamente. El sistema está disponible de forma gratuita e incluye compatibilidad incorporada con una gran cantidad de marcos de aprendizaje automático populares, lo que hace que nuestro enfoque sea útil para resolver problemas prácticos con relativa facilidad y esfuerzo. El uso de la herramienta propuesta en esta investigación permite a los investigadores y profesionales desarrollar rápidamente algoritmos de referencia optimizados en diversos problemas de aprendizaje automático. En algunos escenarios, la solución proporcionada por nuestro sistema podría ser suficiente. Sin embargo, los sistemas AutoML no deben intentar reemplazar a los expertos humanos, sino servir como herramientas complementarias que permitan a los investigadores obtener rápidamente mejores prototipos y conocimientos sobre las estrategias más prometedoras en un problema concreto. Las técnicas de AutoML abren las puertas a revolucionar la forma en que se realiza la investigación y el desarrollo del aprendizaje automático en la academia y la industria.

L'apprentissage automatique (ML) a connu d'énormes succès ces dernières années et repose sur un nombre toujours croissant d'applications réelles. Cependant, la conception d'algorithmes prometteurs pour un problème spécifique nécessite toujours un effort humain considérable. L'apprentissage automatique (AutoML) a pour objectif de sortir l'homme de la boucle. AutoML est généralement traité comme un problème de sélection d’algorithme / hyper-paramètre. Les approches existantes incluent l’optimisation Bayésienne, les algorithmes évolutionnistes et l’apprentissage par renforcement. Parmi eux, auto-sklearn, qui intègre des techniques de meta-learning à l'initialisation de la recherche, occupe toujours une place de choix dans les challenges AutoML. Cette observation a orienté mes recherches vers le domaine du meta-learning. Cette orientation m'a amené à développer un nouveau cadre basé sur les processus de décision Markovien (MDP) et l'apprentissage par renforcement (RL). Après une introduction générale (chapitre 1), mon travail de thèse commence par une analyse approfondie des résultats du Challenge AutoML (chapitre 2). Cette analyse a orienté mon travail vers le meta-learning, menant tout d’abord à proposer une formulation d’AutoML en tant que problème de recommandation, puis à formuler une nouvelle conceptualisation du problème en tant que MDP (chapitre 3). Dans le cadre du MDP, le problème consiste à remplir de manière aussi rapide et efficace que possible une matrice S de meta-learning, dans laquelle les lignes correspondent aux tâches et les colonnes aux algorithmes. Un élément de matrice S (i, j) est la performance de l'algorithme j appliqué à la tâche i. La recherche efficace des meilleures valeurs dans S nous permet d’identifier rapidement les algorithmes les mieux adaptés à des tâches données. Dans le chapitre 4, nous examinons d’abord le cadre classique d’optimisation des hyper-paramètres. Au chapitre 5, une première approche de meta-learning est introduite, qui combine des techniques d'apprentissage actif et de filtrage collaboratif pour prédire les valeurs manquantes dans S. Nos dernières recherches appliquent RL au problème du MDP défini pour apprendre une politique efficace d’exploration de S. Nous appelons cette approche REVEAL et proposons une analogie avec une série de jeux pour permettre de visualiser les stratégies des agents pour révéler progressivement les informations. Cette ligne de recherche est développée au chapitre 6. Les principaux résultats de mon projet de thèse sont : 1) Sélection HP / modèle : j'ai exploré la méthode Freeze-Thaw et optimisé l'algorithme pour entrer dans le premier challenge AutoML, obtenant la 3ème place du tour final (chapitre 3). 2) ActivMetaL : j'ai conçu un nouvel algorithme pour le meta-learning actif (ActivMetaL) et l'ai comparé à d'autres méthodes de base sur des données réelles et artificielles. Cette étude a démontré qu'ActiveMetaL est généralement capable de découvrir le meilleur algorithme plus rapidement que les méthodes de base. 3) REVEAL : j'ai développé une nouvelle conceptualisation du meta-learning en tant que processus de décision Markovien et je l'ai intégrée dans le cadre plus général des jeux REVEAL. Avec un stagiaire en master, j'ai développé des agents qui apprennent (avec l'apprentissage par renforcement) à prédire le meilleur algorithme à essayer. Le travail présenté dans ma thèse est de nature empirique. Plusieurs méta-données du monde réel ont été utilisées dans cette recherche. Des méta-données artificielles et semi-artificielles sont également utilisées dans mon travail. Les résultats indiquent que RL est une approche viable de ce problème, bien qu'il reste encore beaucoup à faire pour optimiser les algorithmes et les faire passer à l’échelle aux problèmes de méta-apprentissage plus vastes. / Machine Learning (ML) has enjoyed huge successes in recent years and an ever- growing number of real-world applications rely on it. However, designing promising algorithms for a specific problem still requires huge human effort. Automated Machine Learning (AutoML) aims at taking the human out of the loop and develop machines that generate / recommend good algorithms for a given ML tasks. AutoML is usually treated as an algorithm / hyper-parameter selection problems, existing approaches include Bayesian optimization, evolutionary algorithms as well as reinforcement learning. Among them, auto-sklearn which incorporates meta-learning techniques in their search initialization, ranks consistently well in AutoML challenges. This observation oriented my research to the Meta-Learning domain. This direction led me to develop a novel framework based on Markov Decision Processes (MDP) and reinforcement learning (RL).After a general introduction (Chapter 1), my thesis work starts with an in-depth analysis of the results of the AutoML challenge (Chapter 2). This analysis oriented my work towards meta-learning, leading me first to propose a formulation of AutoML as a recommendation problem, and ultimately to formulate a novel conceptualisation of the problem as a MDP (Chapter 3). In the MDP setting, the problem is brought back to filling up, as quickly and efficiently as possible, a meta-learning matrix S, in which lines correspond to ML tasks and columns to ML algorithms. A matrix element S(i, j) is the performance of algorithm j applied to task i. Searching efficiently for the best values in S allows us to identify quickly algorithms best suited to given tasks. In Chapter 4 the classical hyper-parameter optimization framework (HyperOpt) is first reviewed. In Chapter 5 a first meta-learning approach is introduced along the lines of our paper ActivMetaL that combines active learning and collaborative filtering techniques to predict the missing values in S. Our latest research applies RL to the MDP problem we defined to learn an efficient policy to explore S. We call this approach REVEAL and propose an analogy with a series of toy games to help visualize agents’ strategies to reveal information progressively, e.g. masked areas of images to be classified, or ship positions in a battleship game. This line of research is developed in Chapter 6. The main results of my PhD project are: 1) HP / model selection: I have explored the Freeze-Thaw method and optimized the algorithm to enter the first AutoML challenge, achieving 3rd place in the final round (Chapter 3). 2) ActivMetaL: I have designed a new algorithm for active meta-learning (ActivMetaL) and compared it with other baseline methods on real-world and artificial data. This study demonstrated that ActiveMetaL is generally able to discover the best algorithm faster than baseline methods. 3) REVEAL: I developed a new conceptualization of meta-learning as a Markov Decision Process and put it into the more general framework of REVEAL games. With a master student intern, I developed agents that learns (with reinforcement learning) to predict the next best algorithm to be tried. To develop this agent, we used surrogate toy tasks of REVEAL games. We then applied our methods to AutoML problems. The work presented in my thesis is empirical in nature. Several real world meta-datasets were used in this research. Artificial and semi-artificial meta-datasets are also used in my work. The results indicate that RL is a viable approach to this problem, although much work remains to be done to optimize algorithms to make them scale to larger meta-learning problems.

Automatic machine learning (AutoML) systems have been shown to perform better when they use metamodels trained offline. Existing offline metalearning approaches treat ML models as black boxes. However, modern ML models often compose multiple ML algorithms into ML pipelines. We expand previous metalearning work on estimating the performance and ranking of ML models by exploiting the metadata about which ML algorithms are used in a given pipeline. We propose a dynamically assembled neural network with the potential to model arbitrary DAG structures. We compare our proposed metamodel against reasonable baselines that exploit varying amounts of pipeline metadata, including metamodels used in existing AutoML systems. We observe that metamodels that fully exploit pipeline metadata are better estimators of pipeline performance. We also find that ranking pipelines based on dataset metafeature similarity outperforms ranking based on performance estimates.

In recent years, the scope of Machine Learning algorithms and its techniques are taking up a notch in every industry (for example, recommendation systems, user behavior analytics, financial applications and many more). In practice, they play an important role in utilizing the power of the vast data we currently generate on a daily basis in our digital world.In this study, we present a comprehensive comparison of different supervised Machine Learning algorithms and feature selection techniques to build a best predictive model as an output. Thus, this predictive model helps companies predict unwanted behavior in their business processes. In addition, we have researched for the automation of all the steps involved (from understanding data to implementing models) in the complete Machine Learning Pipeline, also known as AutoML, and provide a comprehensive survey of the various frameworks introduced in this domain. These frameworks were introduced to solve the problem of CASH (combined algorithm selection and Hyper- parameter optimization), which is basically automation of various pipelines involved in the process of building a Machine Learning predictive model. / Under de senaste åren har omfattningen av maskininlärnings algoritmer och tekniker tagit ett steg i alla branscher (till exempel rekommendationssystem, beteendeanalyser av användare, finansiella applikationer och många fler). I praktiken spelar de en viktig roll för att utnyttja kraften av den enorma mängd data vi för närvarande genererar dagligen i vår digitala värld.I den här studien presenterar vi en omfattande jämförelse av olika övervakade maskininlärnings algoritmer och funktionsvalstekniker för att bygga en bästa förutsägbar modell som en utgång. Således hjälper denna förutsägbara modell företag att förutsäga oönskat beteende i sina affärsprocesser. Dessutom har vi undersökt automatiseringen av alla inblandade steg (från att förstå data till implementeringsmodeller) i den fullständiga maskininlärning rörledningen, även känd som AutoML, och tillhandahåller en omfattande undersökning av de olika ramarna som introducerats i denna domän. Dessa ramar introducerades för att lösa problemet med CASH (kombinerat algoritmval och optimering av Hyper-parameter), vilket i grunden är automatisering av olika rörledningar som är inblandade i processen att bygga en förutsägbar modell för maskininlärning.

With the rapid advancement of hardware and software technologies, machine learning has been pushed to the forefront of business value generating technologies. More and more businesses start to invest in machine learning to keep up with those that have already benefited from it. A local paper processing business is looking to improve upon the estimation of each order's runtime on the machines by leveraging the machine learning technologies. Traditionally, the predictions are done by experienced planners, but the actual runtimes do not always match the predictions. This thesis conducted an investigation about whether a machine learning model could be built to produce better estimations on behalf of the local business. By following a well-defined machine learning workflow in combination with Microsoft's AutoML model builder and data processing techniques, the result shows that predictions made by the machine learning model are able to perform better than the human made ones within an accepted margin.

This study investigates how small and medium sized enterprises (SMEs) and other resource-lacking organisations can utilise automated machine learning (AutoML) to lessen the development hurdles associated with machine learning model development. This is achieved by comparing the performance, cost of usage, as well as usability and documentation for machine learning models developed through two AutoML frameworks: Vertex AI on Google Cloud™ and the open-source library AutoGluon, developed by Amazon Web Services. The study also presents a roadmap and a time plan that can be utilised by resource-lacking enterprises to guide the development of machine learning solutions implemented through AutoML frameworks. The results of the study show that AutoML frameworks are easy to use and capable in generating machine learning models. However, performance is not guaranteed and machine learning projects utilising AutoML frameworks still necessitates substantial development effort. Furthermore, the limiting factor in model performance is often training data quality which AutoML frameworks do not address.

Even a professional photographer sometimes uses automatic default settings that come up with the camera to take a photo. One can debate the quality of outcome from manual vs automatic mode. Until and unless we have a professional level of competence in taking a photo, updating our skills/knowledge as per the latest market trends and having enough time to try out different settings manually, it is worthwhile to use Auto-mode. As camera manufacturers, after several iterations of testing, comes up with the list of ideal parameter values, which is embedded as a factory default setting when we choose auto-mode. For non-professional photographers or amateurs recommend using the auto-mode that comes with the camera for not missing the moment. Similarly, in the context of developing machine learning models, until and unless we have the required data-engineering and ML development competence, time to train and test different ML models and tune different hyper parameter settings, it is worth to try out to Automatic Machine learning feature provided out-of-shelf by all the Cloud-based and Cloud-agnostic ML platforms. This thesis deep dives into evaluating possibility of generating automatic machine learning models with no-code/low-code experience provided by GCP, AWS, Azure and Databricks. We have made a comparison between different ML platforms on generating automatic ML model and presenting the results. It also covers the lessons learnt by developing automatic ML models from a sample dataset across all four ML platforms. Later, we have outlined machine learning subject matter expert’s viewpoints about using Automatic Machine learning models. From this research, we found automatic machine learning can come handy for many off-the-shelf analytical use-cases, this can be highly beneficial especially for time-constrained projects, when resource competence or staffing is a bottleneck and even when competent data scientists want a second-opinion or compare AutoML results with the custom ML model built.