One of the challenges of data mining is finding hyper-parameters for a learning algorithm that will produce the best model for a given dataset. Hyper-parameter optimization automates this process, but it can still take significant time. It has been found that hyperparameter optimization does not always result in induced models with significant improvement over default hyper-parameters, yet no systematic analysis of the role of hyper-parameter optimization in machine learning has been conducted. We propose the use of meta-learning to inform the decision to optimize hyper-parameters based on whether default hyper-parameter performance can be surpassed in a given amount of time. We will build a base of metaknowledge, through a series of experiments, to build predictive models that will assist in the decision process.

One of the most challenging topics of deep learning (DL) is the analysis of temporal series in complex real-world scenarios. The majority of proposed DL methods tend to simplify such environments without considering several factors. The first part of this thesis focuses on developing video surveillance and sports analytic systems, in which obstacles, social interactions, and flow directions are relevant aspects. A DL model is then proposed to predict future paths, taking into account human interactions sharing a common memory, and favouring the most common paths through belief maps. Another model is proposed, adding the possibility to consider agents' goals. This aspect is particularly relevant in sports games where players can share objectives and tactics. Both the proposed models rely on the common hypothesis that the whole amount of labelled data is available from the beginning of the analysis, without evolving. This can be a strong simplification for most real-world scenarios, where data is available as a stream and changes over time. Thus, a theoretical model for continual learning is then developed to face problems where few data come as a stream, and labelling them is a hard task. Finally, continual learning strategies are applied to one of the most challenging scenarios for DL: financial market predictions. A collection of state-of-the-art continual learning techniques are applied to financial indicators representing temporal data. Results achieved during this PhD show how artificial intelligence algorithms can help to solve real-world problems in complex and time-evolving scenarios.

L'apprentissage automatique (ML) a connu d'énormes succès ces dernières années et repose sur un nombre toujours croissant d'applications réelles. Cependant, la conception d'algorithmes prometteurs pour un problème spécifique nécessite toujours un effort humain considérable. L'apprentissage automatique (AutoML) a pour objectif de sortir l'homme de la boucle. AutoML est généralement traité comme un problème de sélection d’algorithme / hyper-paramètre. Les approches existantes incluent l’optimisation Bayésienne, les algorithmes évolutionnistes et l’apprentissage par renforcement. Parmi eux, auto-sklearn, qui intègre des techniques de meta-learning à l'initialisation de la recherche, occupe toujours une place de choix dans les challenges AutoML. Cette observation a orienté mes recherches vers le domaine du meta-learning. Cette orientation m'a amené à développer un nouveau cadre basé sur les processus de décision Markovien (MDP) et l'apprentissage par renforcement (RL). Après une introduction générale (chapitre 1), mon travail de thèse commence par une analyse approfondie des résultats du Challenge AutoML (chapitre 2). Cette analyse a orienté mon travail vers le meta-learning, menant tout d’abord à proposer une formulation d’AutoML en tant que problème de recommandation, puis à formuler une nouvelle conceptualisation du problème en tant que MDP (chapitre 3). Dans le cadre du MDP, le problème consiste à remplir de manière aussi rapide et efficace que possible une matrice S de meta-learning, dans laquelle les lignes correspondent aux tâches et les colonnes aux algorithmes. Un élément de matrice S (i, j) est la performance de l'algorithme j appliqué à la tâche i. La recherche efficace des meilleures valeurs dans S nous permet d’identifier rapidement les algorithmes les mieux adaptés à des tâches données. Dans le chapitre 4, nous examinons d’abord le cadre classique d’optimisation des hyper-paramètres. Au chapitre 5, une première approche de meta-learning est introduite, qui combine des techniques d'apprentissage actif et de filtrage collaboratif pour prédire les valeurs manquantes dans S. Nos dernières recherches appliquent RL au problème du MDP défini pour apprendre une politique efficace d’exploration de S. Nous appelons cette approche REVEAL et proposons une analogie avec une série de jeux pour permettre de visualiser les stratégies des agents pour révéler progressivement les informations. Cette ligne de recherche est développée au chapitre 6. Les principaux résultats de mon projet de thèse sont : 1) Sélection HP / modèle : j'ai exploré la méthode Freeze-Thaw et optimisé l'algorithme pour entrer dans le premier challenge AutoML, obtenant la 3ème place du tour final (chapitre 3). 2) ActivMetaL : j'ai conçu un nouvel algorithme pour le meta-learning actif (ActivMetaL) et l'ai comparé à d'autres méthodes de base sur des données réelles et artificielles. Cette étude a démontré qu'ActiveMetaL est généralement capable de découvrir le meilleur algorithme plus rapidement que les méthodes de base. 3) REVEAL : j'ai développé une nouvelle conceptualisation du meta-learning en tant que processus de décision Markovien et je l'ai intégrée dans le cadre plus général des jeux REVEAL. Avec un stagiaire en master, j'ai développé des agents qui apprennent (avec l'apprentissage par renforcement) à prédire le meilleur algorithme à essayer. Le travail présenté dans ma thèse est de nature empirique. Plusieurs méta-données du monde réel ont été utilisées dans cette recherche. Des méta-données artificielles et semi-artificielles sont également utilisées dans mon travail. Les résultats indiquent que RL est une approche viable de ce problème, bien qu'il reste encore beaucoup à faire pour optimiser les algorithmes et les faire passer à l’échelle aux problèmes de méta-apprentissage plus vastes. / Machine Learning (ML) has enjoyed huge successes in recent years and an ever- growing number of real-world applications rely on it. However, designing promising algorithms for a specific problem still requires huge human effort. Automated Machine Learning (AutoML) aims at taking the human out of the loop and develop machines that generate / recommend good algorithms for a given ML tasks. AutoML is usually treated as an algorithm / hyper-parameter selection problems, existing approaches include Bayesian optimization, evolutionary algorithms as well as reinforcement learning. Among them, auto-sklearn which incorporates meta-learning techniques in their search initialization, ranks consistently well in AutoML challenges. This observation oriented my research to the Meta-Learning domain. This direction led me to develop a novel framework based on Markov Decision Processes (MDP) and reinforcement learning (RL).After a general introduction (Chapter 1), my thesis work starts with an in-depth analysis of the results of the AutoML challenge (Chapter 2). This analysis oriented my work towards meta-learning, leading me first to propose a formulation of AutoML as a recommendation problem, and ultimately to formulate a novel conceptualisation of the problem as a MDP (Chapter 3). In the MDP setting, the problem is brought back to filling up, as quickly and efficiently as possible, a meta-learning matrix S, in which lines correspond to ML tasks and columns to ML algorithms. A matrix element S(i, j) is the performance of algorithm j applied to task i. Searching efficiently for the best values in S allows us to identify quickly algorithms best suited to given tasks. In Chapter 4 the classical hyper-parameter optimization framework (HyperOpt) is first reviewed. In Chapter 5 a first meta-learning approach is introduced along the lines of our paper ActivMetaL that combines active learning and collaborative filtering techniques to predict the missing values in S. Our latest research applies RL to the MDP problem we defined to learn an efficient policy to explore S. We call this approach REVEAL and propose an analogy with a series of toy games to help visualize agents’ strategies to reveal information progressively, e.g. masked areas of images to be classified, or ship positions in a battleship game. This line of research is developed in Chapter 6. The main results of my PhD project are: 1) HP / model selection: I have explored the Freeze-Thaw method and optimized the algorithm to enter the first AutoML challenge, achieving 3rd place in the final round (Chapter 3). 2) ActivMetaL: I have designed a new algorithm for active meta-learning (ActivMetaL) and compared it with other baseline methods on real-world and artificial data. This study demonstrated that ActiveMetaL is generally able to discover the best algorithm faster than baseline methods. 3) REVEAL: I developed a new conceptualization of meta-learning as a Markov Decision Process and put it into the more general framework of REVEAL games. With a master student intern, I developed agents that learns (with reinforcement learning) to predict the next best algorithm to be tried. To develop this agent, we used surrogate toy tasks of REVEAL games. We then applied our methods to AutoML problems. The work presented in my thesis is empirical in nature. Several real world meta-datasets were used in this research. Artificial and semi-artificial meta-datasets are also used in my work. The results indicate that RL is a viable approach to this problem, although much work remains to be done to optimize algorithms to make them scale to larger meta-learning problems.

The following thesis explores the impact of the dataset distributional prop- erties on classification performance. We use Gaussian copulas to generate 1000 artificial dataset and train classifiers on them. We train Generalized linear models, Distributed Random forest, Extremely randomized trees and Gradient boosting machines via H2O.ai machine learning platform accessed by R. Classi- fication performance on these datasets is evaluated and empirical observations on influence are presented. Secondly, we use real Australian credit dataset and predict which classifier is possibly going to work best. The predicted perfor- mance for any individual method is based on penalizing the differences between the Australian dataset and artificial datasets where the method performed com- paratively better, but it failed to predict correctly. 1

Earthquake detection is an important task, focusing on detecting seismic events in past data or in real time from seismic time series. In the past few decades, due to the increasing amount of available seismic data, research in seismic event detection shows remarkable success using neural networks and other machine learning techniques. However, creating high quality labeled data sets is still a manual process that demands tremendous amount of time and expert knowledge, and is stifling big data innovation. When compiling a data set, it is unclear how many earthquakes and noise are mislabeled. Another challenge is how to promote the general applicability of the machine learning based models to different geographical regions. The models trained by data sets from one location should be applicable to the detection at other locations. This thesis explores the most popular deep learning model, convolutional neural networks (CNN), to build a single location detection model. In addition, we build more robust generalized earthquake detection models using transfer learning and meta learning. We also introduce a process for generating high quality labeled datasets. Our technique achieves high detection accuracy even on low signal to noise ratio events. The AI techniques explored in this research have potential to be transferred to other domains that utilize signal processing. There are a myriad of potential applications, with audio processing probably being one of the most directly relevant. Any field that deals with waveforms (e.g. seismic, audio, light) can utilize the developed techniques.

Collecting a sufficient amount of data and centralizing them are both costly and privacy-concerning operations. These practical concerns arise due to the communication costs between data collecting devices and data being personal such as text messages of an end user. The goal is to train generalizable machine learning models with constraints on data without sharing or transferring the data. In this thesis, we will present solutions to several aspects of learning with data constraints, such as processing and supervision. We focus on federated learning, online learning, and learning generalizable representations and provide setting-specific training recipes. In the first scenario, we tackle a federated learning problem where data is decentralized through different users and should not be centralized. Traditional approaches either ignore the heterogeneity problem or increase communication costs to handle it. Our solution carefully addresses the heterogeneity issue of user data by imposing a dynamic regularizer that adapts to the heterogeneity of each user without extra transmission costs. Theoretically, we establish convergence guarantees. We extend our ideas to personalized federated learning, where the model is customized to each end user, and heterogeneous federated learning, where users support different model architectures. As a next scenario, we consider online meta-learning, where there is only one user, and the data distribution of the user changes over time. The goal is to adapt new data distributions with very few labeled data from each distribution. A naive way is to store data from different distributions to train a model from scratch with sufficient data. Our solution efficiently summarizes the information from each task data so that the memory footprint does not scale with the number of tasks. Lastly, we aim to train generalizable representations given a dataset. We consider a setting where we have access to a powerful teacher (more complex) model. Traditional methods do not distinguish points and force the model to learn all the information from the powerful model. Our proposed method focuses on the learnable input space and carefully distills attainable information from the teacher model by discarding the over-capacity information. We compare our methods with state-of-the-art methods in each setup and show significant performance improvements. Finally, we discuss potential directions for future work.

The performance of classification algorithms is mainly governed by the hyperparameter configurations deployed. Traditional search-based algorithms tend to require extensive hyperparameter evaluations to select the desirable configurations during the process, and they are often very inefficient for implementations on large-scale tasks. In this dissertation, we resort to solving the problem of hyperparameter selection via meta-learning which provides a mechanism that automatically recommends the promising ones without any inefficient evaluations. In its approach, a meta-learner is constructed on the metadata extracted from historical classification problems which directly determines the success of recommendations. Designing fine meta-learners to recommend effective hyperparameter configurations efficiently is of practical importance. This dissertation divides into six chapters: the first chapter presents the research background and related work, the second to the fifth chapters detail our main work and contributions, and the sixth chapter concludes the dissertation and pictures our possible future work. In the second and third chapters, we propose two (kernel) multivariate sparse-group Lasso (SGLasso) approaches for automatic meta-feature selection. Previously, meta-features were usually picked by researchers manually based on their preferences and experience or by wrapper method, which is either less effective or time-consuming. SGLasso, as an embedded feature selection model, can select the most effective meta-features during the meta-learner training and thus guarantee the optimality of both meta-features and meta-learner which are essential for successful recommendations. In the fourth chapter, we formulate the problem of hyperparameter recommendation as a problem of low-rank tensor completion. The hyperparameter search space was often stretched to a one-dimensional vector, which removes the spatial structure of the search space and ignores the correlations that existed between the adjacent hyperparameters and these characteristics are crucial in meta-learning. Our contributions are to instantiate the search space of hyperparameters as a multi-dimensional tensor and develop a novel kernel tensor completion algorithm that is applied to estimate the performance of hyperparameter configurations. In the fifth chapter, we propose to learn the latent features of performance space via denoising autoencoders. Although the search space is usually high-dimensional, the performance of hyperparameter configurations is usually correlated to each other to a certain degree and its main structure lies in a much lower-dimensional manifold that describes the performance distribution of the search space. Denoising autoencoders are applied to extract the latent features on which two effective recommendation strategies are built. Extensive experiments are conducted to verify the effectiveness of our proposed approaches, and various empirical outcomes have shown that our approaches can recommend promising hyperparameters for real problems and significantly outperform the state-of-the-art meta-learning-based methods as well as search algorithms such as random search, Bayesian optimization, and Hyperband.

With the development of networked systems, vulnerabilities underlying a network have kept increasing in recent years, and cyber security has become an essential part when building such networks. One of the most popular methods of evaluating the security of a network is penetration testing. However, we have seen a shortage of experts in the penetration testing field due to its complexity and the training cost. One way to alleviate this problem is automated penetration testing, which automates the penetration test process using algorithms, including the Attack Graph model, Partially Observable Markov Decision Process (POMDP) method. In this thesis, we demonstrates the application of reinforcement learning algorithms on penetration testing problems and shows the potential application of meta-learning methods on such problems to boost the generalization ability of reinforcement learning algorithms. We first test the performance of Advantage Actor Critic (A2C) and Double Deep Q-Network (DDQN) on different static networks and compare their convergence speed, stability and total rewards achieved. Then we incorporate meta-learning ideas into reinforcement learning algorithms and propose a new model named MetaNet. Our results show that reinforcement learning algorithms are capable of solving penetration testing problems with little prior knowledge, and by using meta-learning methods, MetaNet shows a great improvement in generalization ability. To conduct our experiments, we first create a test environment, which is a structured network mimicking actual communication networks in real-world. Each network is composed of several hosts, and each host contains several services that can be compromised. Then we apply A2C and DDQN on these networks. The algorithms start from a certain host and try to compromise the target host. Our results show that both A2C and DDQN are capable to compromise the target host and achieve positive rewards under most circumstances. To increase the generalization ability of these algorithms, we propose MetaNet, where we add additional inputs to the model, wrap the model with Long Short-Term Memory (LSTM) and train the model on different networks at once. Our results show that MetaNet not only keeps high winning ratios on networks that it is trained on but also performs better than the vanilla algorithms on other unseen networks. / I och med utvecklingen av nätverkssystem har sårbarheterna i ett nätverk ökat under de senaste åren, och cybersäkerhet har blivit en viktig del när man bygger sådana nätverk. En av de mest populära metoderna för att utvärdera säkerheten i ett nätverk är penetrationstestning. Vi har dock sett en brist på experter inom penetrationstestområdet på grund av dess komplexitet och utbildningskostnaden. Ett sätt att lindra detta problem är automatiserad penetrationstestning, som automatiserar penetrationstestprocessen med hjälp av algoritmer, inklusive Attack Graph-modellen, POMDP-metoden. Denna avhandling demonstrerar tillämpningen av förstärkningsinlärningsalgoritmer på problem med penetrationstestning och visar den potentiella tillämpningen av meta-inlärningsmetoder på sådana problem för att öka generaliseringsförmågan hos förstärkningsinlärningsalgoritmer. Vi testar först prestandan hos A2C och DDQN på olika statiska nätverk och jämför deras konvergenshastighet, stabilitet och totala uppnådda belöningar. Sedan införlivar vi meta-lärande idéer i förstärkningsinlärningsalgoritmer och föreslår en ny modell som heter MetaNet. Våra resultat visar att förstärkningsinlärningsalgoritmer kan lösa penetrationstestningsproblem med få förkunskaper, och genom att använda meta-inlärningsmetoder visar MetaNet en stor förbättring av generaliseringsförmågan. För att genomföra våra experiment skapar vi först en testmiljö, som är ett strukturerat nätverk som efterliknar faktiska kommunikationsnätverk i verkligheten. Varje nätverk består av flera värdar, och varje värd innehåller flera tjänster som kan äventyras. Sedan tillämpar vi A2C och DDQN på dessa nätverk. Algoritmerna utgår från en viss värd och försöker äventyra målvärden. Våra resultat visar att både A2C och DDQN är kapabla att äventyra målvärden och uppnå positiva belöningar under de flesta omständigheter. För att öka generaliseringsförmågan hos dessa algoritmer föreslår vi MetaNet, där vi lägger till ytterligare input till modellen, slår in modellen med LSTM och tränar modellen på olika nätverk samtidigt. Våra resultat visar att MetaNet inte bara håller höga vinstkvoter på nätverk som det är tränade på utan också presterar bättre än vaniljalgoritmerna på andra osynliga nätverk.

Managing a system of network services is a complex and large-scale task that often lacks a trivial optimal solution. Deep Reinforcement Learning (RL) has shown great potential in being able to solve these tasks in static settings. However, in practice, the RL agents struggle to generalize their control policies enough to work in more dynamic real-world environments. To achieve a generality between environments, multiple contributions are made by this thesis. Low-level metrics are collected from each node in the system to help explain changes in the end-to-end delay of the system. To achieve generality in its control policy, more ways to observe and understand the dynamic environment and how it changes are provided to the RL agent by introducing the end-to-end delay of each service in the system to its observation space. Another approach to achieving more generality in RL policies is Model-Agnostic Meta-Learning (MAML), a type of Meta-Learning approach where instead of learning to solve a specific task, the model learns to learn how to quickly solve a new task based on prior knowledge. Results show that low-level metrics yield a much greater generality when helping to explain the delay of a system. Applying MAML to the problem is beneficial in adding generality to a learned RL policy and making the adaptation to a new task faster. If the RL agent can observe the changes to the underlying dynamics of the environment between tasks by itself, the policy can achieve this generality by itself without the need for a more complex method. However, if acquiring or observing the necessary data is too expensive or complex, switching to a Meta-Learning approach might be beneficial to increase generality. / Hanteringen av ett system med nätverksstjänster är ett komplext och stor skaligt problem där den optimal lösning inte är trivial. Djup förstärkningsinlärning har visat stor potential i att kunna lösa dessa problem i statiska miljöer. Dock är det svårt att generalisera lösningarna tillräckligt för att fungera i mer komplicerade och realistiska dynamiska miljöer. För att uppnå mer generella lösningar mellan miljöer presenterar denna masteruppsats flera möjliga lösningar. Lågnivåmetrik samlas in från varje nod i systemet för att hjälpa förklara skillnader i den totala responstiden för varje tjänst i systemet. För att generalisera förstärkningsinlärningsmodellen kan den förses med fler sätt att observera miljön, och därmed lära sig förstå hur den dynamiska miljön förändras. En annan metod för att uppnå mer generalitet inom förstärkningsinlärning är Model-Agnostic Meta-Learning (MAML), en typ av Meta-Learning där istället för att lära sig att lösa en specifik uppgift, modellen lär sig att lära sig att snabbt lösa en ny uppgift baserat på sin tidigare kunskap. Resultaten visar att lågnivåmetriken leder till en mycket högre generalitet i att hjälpa till att förklara responstiden av ett system. Att applicera MAML till problemet hjälper att bidra med generalitet till en förstärkningsinlärningsmodell och gör anpassningen till en ny uppgift snabbare. Om modellen själv kan observera ändringarna i underliggande dynamiken bakom miljön mellan uppgifter kan den uppnå mer generalitet utan ett behov av en mer komplex metod som MAML. Däremot, om det är svårt eller dyrt att få tag på eller observera den nödvändiga datan, kan ett byte till en Meta-Learning baserad metod vara mer fördelaktig för att öka generaliteten.

A área de aprendizado de máquina passa por uma grande expansão em seu universo de aplicações. Algoritmos de indução de modelos preditivos têm sido responsáveis pela realização de tarefas que eram inviáveis ou consideradas exclusividade do campo de ação humano até recentemente. Contudo, ainda é necessária a supervisão humana durante a construção de conjuntos de treinamento, como é o caso da tarefa de classificação. Tal construção se dá por meio da rotulação manual de cada exemplo, atribuindo a ele pelo menos uma classe. Esse processo, por ser manual, pode ter um custo elevado se for necessário muitas vezes. Uma técnica sob investigação corrente, capaz de mitigar custos de rotulação, é o aprendizado ativo. Dado um orçamento limitado, o objetivo de uma estratégia de amostragem ativa é direcionar o esforço de treinamento para os exemplos essenciais. Existem diversas abordagens efetivas de selecionar ativamente os exemplos mais importantes para consulta ao supervisor. Entretanto, não é possível, sem incorrer em custos adicionais, testá-las de antemão quanto à sua efetividade numa dada aplicação. Ainda mais crítica é a necessidade de que seja escolhido um algoritmo de aprendizado para integrar a estratégia de aprendizado ativo antes que se disponha de um conjunto de treinamento completo. Para lidar com esses desafios, esta tese apresenta como principais contribuições: uma estratégia baseada na inibição do algoritmo de aprendizado nos momentos menos propícios ao seu funcionamento; e, a experimentação da seleção de algoritmos de aprendizado, estratégias ativas de consulta ou pares estratégia-algoritmo baseada em meta-aprendizado, visando a experimentação de formas de escolha antes e durante o processo de rotulação. A estratégia de amostragem proposta é demonstrada competitiva empiricamente. Adicionalmente, experimentos iniciais com meta-aprendizado indicam a possibilidade de sua aplicação em aprendizado ativo, embora tenha sido identificado que investigações mais extensivas e aprofundadas sejam necessárias para apurar sua real efetividade prática. Importantes contribuições metodológicas são descritas neste documento, incluindo uma análise frequentemente negligenciada pela literatura da área: o risco devido à variabilidade dos algoritmos. Por fim, são propostas as curvas e faixas de ranqueamento, capazes de sumarizar, num único gráfico, experimentos de uma grande coleção de conjuntos de dados. / The machine learning area undergoes a major expansion in its universe of applications. Algorithms for the induction of predictive models have made it possible to carry out tasks that were once considered unfeasible or restricted to be solved by humans. However, human supervision is still needed to build training sets, for instance, in the classification task. Such building is usually performed by manual labeling of each instance, providing it, at least, one class. This process has a high cost due to its manual nature. A current technique under research, able to mitigate labeling costs, is called active learning. The goal of an active learning strategy is to manage the training effort to focus on the most relevant instances, within a budget. Several effective sampling approaches having been proposed. However, when one needs to choose the proper strategy for a given problem, they are impossible to test beforehand without incurring into additional costs. Even more critical is the need to choose a learning algorithm to integrate the active learning strategy before the existence of a complete training set. This thesis presents two major contributions to cope with such challenges: a strategy based on the learning algorithm inhibition when it is prone to inaccurate predictions; and, an attempt to automatically select the learning algorithms, active querying strategies or pairs strategy-algorithm, based on meta-learning. This attempt tries to verify the feasibility of such kind of decision making before and during the learning process. The proposed sampling approach is empirically shown to be competitive. Additionally, meta-learning experiments show that it can be applied to active learning, although more a extensive investigation is still needed to assess its real practical effectivity. Important methodological contributions are made in this document, including an often neglected analysis in the literature of active learning: the risk due to the algorithms variability. A major methodological contribution, called ranking curves, is presented.