L'émergence du phénomène Big Data a créé un besoin grandissant en analyse de données, mais, bien souvent, cette analyse est conduite par des experts de différents domaines ayant peu d'expérience en science des données. On s'intéresse donc à ce besoin d'assistance à l'analyse de données, qui commence tout juste à recevoir une certaine attention des communautés scientifiques, donnant naissance au domaine de la méta-analyse. Les premières approches du sujet se révélant souvent similaires et peu abouties, on tente en particulier de permettre de nouvelles approches de méta-analyse pour adresser ce problème d'assistance à l'analyse de données. Pour ce faire, une première étape cruciale est de déterminer ce qu'est une méta-analyse performante, aucun standard n'ayant encore été établi dans ce domaine relativement neuf. On propose ainsi un cadre générique d'évaluation de méta-analyse, permettant de comparer et caractériser finement diverses techniques de méta- analyse. Ensuite, afin d'ouvrir de nouvelles voies, on s'intéresse à un verrou majeur de la méta-analyse : la caractérisation de jeu de données. On propose et évalue alors une caractérisation par dissimilarité faisant usage de toute l'information disponible pour autoriser de nouvelles approches de méta-analyse. L'utilisation de cette caractérisation par dissimilarité permettant de recommander facilement des processus d'analyse de données complets, on décrit enfin les nouvelles approches de méta-analyses rendues possibles, ainsi que les processus afférents d'assistance à l'analyse de données. / The emergence of the big data phenomenon has led to increasing demands in data analysis, which most often are conducted by other domains experts with little experience in data science. We then consider this important demand in intelligent assistance to data analysis, which receives an increasing attention from the scientific community. The first takes on the subject often possessing similar shortcomings, we propose to address it through new processes of meta-analysis. No evaluation standard having yet been set in this relatively new domain, we first propose a meta-analysis evaluation framework that will allow us to test and compare the developed methods. In order to open new approaches of meta-analysis, we then consider one of its recurring issue: dataset characterization. We then propose and evaluate such a characterization, consisting in a dissimilarity between datasets making use of a precise topological description to compare them. This dissimilarity allows a new meta-analysis approach producing recommendations of complete data analysis processes, which we then evaluate on a proof of concept. We thus detail the proposed methods of meta-analysis, and the associated process of assistance to data analysis.

In many areas of knowledge, considerable amounts of time have been spent to comprehend and to treat noisy data, one of the most common problems regarding information collection, transmission and storage. These noisy data, when used for training Machine Learning techniques, lead to increased complexity in the induced classification models, higher processing time and reduced predictive power. Treating them in a preprocessing step may improve the data quality and the comprehension of the problem. This Thesis aims to investigate the use of data complexity measures capable to characterize the presence of noise in datasets, to develop new efficient noise ltering techniques in such subsamples of problems of noise identification compared to the state of art and to recommend the most properly suited techniques or ensembles for a specific dataset by meta-learning. Both artificial and real problem datasets were used in the experimental part of this work. They were obtained from public data repositories and a cooperation project. The evaluation was made through the analysis of the effect of artificially generated noise and also by the feedback of a domain expert. The reported experimental results show that the investigated proposals are promising. / Em diversas áreas do conhecimento, um tempo considerável tem sido gasto na compreensão e tratamento de dados ruidosos. Trata-se de uma ocorrência comum quando nos referimos a coleta, a transmissão e ao armazenamento de informações. Esses dados ruidosos, quando utilizados na indução de classificadores por técnicas de Aprendizado de Maquina, aumentam a complexidade da hipótese obtida, bem como o aumento do seu tempo de indução, além de prejudicar sua acurácia preditiva. Trata-los na etapa de pré-processamento pode significar uma melhora da qualidade dos dados e um aumento na compreensão do problema estudado. Esta Tese investiga medidas de complexidade capazes de caracterizar a presença de ruídos em um conjunto de dados, desenvolve novos filtros que sejam mais eficientes em determinados nichos do problema de detecção e remoção de ruídos que as técnicas consideradas estado da arte e recomenda as mais apropriadas técnicas ou comitês de técnicas para um determinado conjunto de dados por meio de meta-aprendizado. As bases de dados utilizadas nos experimentos realizados neste trabalho são tanto artificiais quanto reais, coletadas de repositórios públicos e fornecidas por projetos de cooperação. A avaliação consiste tanto da adição de ruídos artificiais quanto da validação de um especialista. Experimentos realizados mostraram o potencial das propostas investigadas.

A área de aprendizado de máquina passa por uma grande expansão em seu universo de aplicações. Algoritmos de indução de modelos preditivos têm sido responsáveis pela realização de tarefas que eram inviáveis ou consideradas exclusividade do campo de ação humano até recentemente. Contudo, ainda é necessária a supervisão humana durante a construção de conjuntos de treinamento, como é o caso da tarefa de classificação. Tal construção se dá por meio da rotulação manual de cada exemplo, atribuindo a ele pelo menos uma classe. Esse processo, por ser manual, pode ter um custo elevado se for necessário muitas vezes. Uma técnica sob investigação corrente, capaz de mitigar custos de rotulação, é o aprendizado ativo. Dado um orçamento limitado, o objetivo de uma estratégia de amostragem ativa é direcionar o esforço de treinamento para os exemplos essenciais. Existem diversas abordagens efetivas de selecionar ativamente os exemplos mais importantes para consulta ao supervisor. Entretanto, não é possível, sem incorrer em custos adicionais, testá-las de antemão quanto à sua efetividade numa dada aplicação. Ainda mais crítica é a necessidade de que seja escolhido um algoritmo de aprendizado para integrar a estratégia de aprendizado ativo antes que se disponha de um conjunto de treinamento completo. Para lidar com esses desafios, esta tese apresenta como principais contribuições: uma estratégia baseada na inibição do algoritmo de aprendizado nos momentos menos propícios ao seu funcionamento; e, a experimentação da seleção de algoritmos de aprendizado, estratégias ativas de consulta ou pares estratégia-algoritmo baseada em meta-aprendizado, visando a experimentação de formas de escolha antes e durante o processo de rotulação. A estratégia de amostragem proposta é demonstrada competitiva empiricamente. Adicionalmente, experimentos iniciais com meta-aprendizado indicam a possibilidade de sua aplicação em aprendizado ativo, embora tenha sido identificado que investigações mais extensivas e aprofundadas sejam necessárias para apurar sua real efetividade prática. Importantes contribuições metodológicas são descritas neste documento, incluindo uma análise frequentemente negligenciada pela literatura da área: o risco devido à variabilidade dos algoritmos. Por fim, são propostas as curvas e faixas de ranqueamento, capazes de sumarizar, num único gráfico, experimentos de uma grande coleção de conjuntos de dados. / The machine learning area undergoes a major expansion in its universe of applications. Algorithms for the induction of predictive models have made it possible to carry out tasks that were once considered unfeasible or restricted to be solved by humans. However, human supervision is still needed to build training sets, for instance, in the classification task. Such building is usually performed by manual labeling of each instance, providing it, at least, one class. This process has a high cost due to its manual nature. A current technique under research, able to mitigate labeling costs, is called active learning. The goal of an active learning strategy is to manage the training effort to focus on the most relevant instances, within a budget. Several effective sampling approaches having been proposed. However, when one needs to choose the proper strategy for a given problem, they are impossible to test beforehand without incurring into additional costs. Even more critical is the need to choose a learning algorithm to integrate the active learning strategy before the existence of a complete training set. This thesis presents two major contributions to cope with such challenges: a strategy based on the learning algorithm inhibition when it is prone to inaccurate predictions; and, an attempt to automatically select the learning algorithms, active querying strategies or pairs strategy-algorithm, based on meta-learning. This attempt tries to verify the feasibility of such kind of decision making before and during the learning process. The proposed sampling approach is empirically shown to be competitive. Additionally, meta-learning experiments show that it can be applied to active learning, although more a extensive investigation is still needed to assess its real practical effectivity. Important methodological contributions are made in this document, including an often neglected analysis in the literature of active learning: the risk due to the algorithms variability. A major methodological contribution, called ranking curves, is presented.

Machine learning solutions have been successfully used to solve many simple and complex problems. However, their development process still relies on human experts to perform tasks such as data preprocessing, feature engineering and model selection. As the complexity of these tasks increases, so does the demand for automated solutions, namely Automated Machine Learning (AutoML). Most algorithms employed in these systems have hyperparameters whose configuration may directly affect their predictive performance. Therefore, hyperparameter tuning is a recurring task in AutoML systems. This thesis investigated how to efficiently automate hyperparameter tuning by means of Meta-learning. To this end, large-scale experiments were performed tuning the hyperparameters of different classification algorithms, and an enhanced experimental methodology was adopted throughout the thesis to explore and learn the hyperparameter profiles for different classification algorithms. The results also showed that in many cases the default hyperparameter settings induced models that are on par with those obtained by tuning. Hence, a new Meta-learning recommender system was proposed to identify when it is better to use default values and when to tune classification algorithms for each new dataset. The proposed system is capable of generalizing several learning processes into a single modular framework, along with the possibility of assigning different algorithms. Furthermore, a descriptive analysis of model predictions is used to identify which data characteristics affect the necessity for tuning in each one of the algorithms investigated in the thesis. Experimental results also demonstrated that the proposed recommender system reduced the time spent on optimization processes, without reducing the predictive performance of the induced models. Depending on the target algorithm, the Meta-learning recommender system can statistically outperform the baselines. The significance of these results opens a number of new avenues for future work. / Soluções de aprendizado de máquina tem sido cada vez mais usadas com sucesso para resolver problemas dos mais simples aos complexos. Entretanto, o processo de desenvolvimento de tais soluções ainda é um processo que depende da ação de especialistas humanos em tarefas como: pré-processamento dos dados, engenharia de features e seleção de modelos. Consequentemente, quando a complexidade destas tarefas atinge um nível muito alto, há a necessidade de soluções automatizadas, denominadas por Aprendizado de Máquina automatizado (AutoML). A maioria dos algoritmos usados em tais sistemas possuem hiper-parâmetros cujos valores podem afetar diretamente o desempenho preditivo dos modelos gerados. Assim sendo, o ajuste de hiper-parâmetros é uma tarefa recorrente no desenvolvimento de sistems de AutoML. Nesta tese investigou-se a automatização do ajuste de hiper-parâmetros por meio de Meta-aprendizado. Seguindo essa linha, experimentos massivos foram realizados para ajustar os hiper-parâmetros de diferentes algoritmos de classificação. Além disso, uma metodologia experimental aprimorada e adotada ao lngo da tese perimtiu identificar diferentes perfis de ajuste para diferentes algoritmos de classificação. Entretanto, os resultados também mostraram que em muitos casos as configurações default destes algoritmos induziram modelos mais precisos do que os obtidos por meio de ajuste. Assim, foi proposto um novo sistema de recomendação baseado em Meta-learning para identificar quando é melhor realizar o ajuste de parâmetros para os algoritmos de classificação ou apenas usar os valores default. O sistema proposto é capaz de generalizar várias etapas do aprendizado em um único framework modular, juntamente com a possibilidade de avaliar diferentes algoritmos de aprendizado de máquina. As análises descritivas das predições obtidas pelo sistema indicaram quais características podem ser responsáveis por determinar quando o ajuste se faz necessário para cada um dos algoritmos investigados na tese. Os resultados também demonstraram que o sistema recomendador proposto reduziu o tempo gasto com a otimização mantendo o desempenho preditivo dos modelos gerados. Além disso, dependendo do algoritmo de classificação modelado, o sistema foi estatisticamente superior aos baselines. A significância desdes resultados abre um novo número de oportunidades para trabalhos futuros.

We propose a new ensemble algorithm: the meta-boosting algorithm. This algorithm enables the original Adaboost algorithm to improve the decisions made by different WeakLearners utilizing the meta-learning approach. Better accuracy results are achieved since this algorithm reduces both bias and variance. However, higher accuracy also brings higher computational complexity, especially on big data. We then propose the parallelized meta-boosting algorithm: Parallelized-Meta-Learning (PML) using the MapReduce programming paradigm on Hadoop. The experimental results on the Amazon EC2 cloud computing infrastructure show that PML reduces the computation complexity enormously while retaining lower error rates than the results on a single computer. As we know MapReduce has its inherent weakness that it cannot directly support iterations in an algorithm, our approach is a win-win method, since it not only overcomes this weakness, but also secures good accuracy performance. The comparison between this approach and a contemporary algorithm AdaBoost.PL is also performed.

One of the issues facing credit card fraud detection systems is that a significant percentage of transactions labeled as fraudulent are in fact legitimate. These “false alarms” delay the detection of fraudulent transactions. Analysis of 11 months of credit card transaction data from a major Canadian bank was conducted to determine savings improvements that can be achieved by identifying truly fraudulent transactions. A meta-classifier model was used in this research. This model consists of 3 base classifiers constructed using the k-nearest neighbour, decision tree, and naïve Bayesian algorithms. The naïve Bayesian algorithm was also used as the meta-level algorithm to combine the base classifier predictions to produce the final classifier. Results from this research show that when a meta-classifier was deployed in series with the Bank’s existing fraud detection algorithm a 24% to 34% performance improvement was achieved resulting in $1.8 to $2.6 million cost savings per year.

One of the issues facing credit card fraud detection systems is that a significant percentage of transactions labeled as fraudulent are in fact legitimate. These “false alarms” delay the detection of fraudulent transactions. Analysis of 11 months of credit card transaction data from a major Canadian bank was conducted to determine savings improvements that can be achieved by identifying truly fraudulent transactions. A meta-classifier model was used in this research. This model consists of 3 base classifiers constructed using the k-nearest neighbour, decision tree, and naïve Bayesian algorithms. The naïve Bayesian algorithm was also used as the meta-level algorithm to combine the base classifier predictions to produce the final classifier. Results from this research show that when a meta-classifier was deployed in series with the Bank’s existing fraud detection algorithm a 24% to 34% performance improvement was achieved resulting in $1.8 to $2.6 million cost savings per year.

abstract: Buildings consume nearly 50% of the total energy in the United States, which drives the need to develop high-fidelity models for building energy systems. Extensive methods and techniques have been developed, studied, and applied to building energy simulation and forecasting, while most of work have focused on developing dedicated modeling approach for generic buildings. In this study, an integrated computationally efficient and high-fidelity building energy modeling framework is proposed, with the concentration on developing a generalized modeling approach for various types of buildings. First, a number of data-driven simulation models are reviewed and assessed on various types of computationally expensive simulation problems. Motivated by the conclusion that no model outperforms others if amortized over diverse problems, a meta-learning based recommendation system for data-driven simulation modeling is proposed. To test the feasibility of the proposed framework on the building energy system, an extended application of the recommendation system for short-term building energy forecasting is deployed on various buildings. Finally, Kalman filter-based data fusion technique is incorporated into the building recommendation system for on-line energy forecasting. Data fusion enables model calibration to update the state estimation in real-time, which filters out the noise and renders more accurate energy forecast. The framework is composed of two modules: off-line model recommendation module and on-line model calibration module. Specifically, the off-line model recommendation module includes 6 widely used data-driven simulation models, which are ranked by meta-learning recommendation system for off-line energy modeling on a given building scenario. Only a selective set of building physical and operational characteristic features is needed to complete the recommendation task. The on-line calibration module effectively addresses system uncertainties, where data fusion on off-line model is applied based on system identification and Kalman filtering methods. The developed data-driven modeling framework is validated on various genres of buildings, and the experimental results demonstrate desired performance on building energy forecasting in terms of accuracy and computational efficiency. The framework could be easily implemented into building energy model predictive control (MPC), demand response (DR) analysis and real-time operation decision support systems. / Dissertation/Thesis / Doctoral Dissertation Industrial Engineering 2016

We propose a new ensemble algorithm: the meta-boosting algorithm. This algorithm enables the original Adaboost algorithm to improve the decisions made by different WeakLearners utilizing the meta-learning approach. Better accuracy results are achieved since this algorithm reduces both bias and variance. However, higher accuracy also brings higher computational complexity, especially on big data. We then propose the parallelized meta-boosting algorithm: Parallelized-Meta-Learning (PML) using the MapReduce programming paradigm on Hadoop. The experimental results on the Amazon EC2 cloud computing infrastructure show that PML reduces the computation complexity enormously while retaining lower error rates than the results on a single computer. As we know MapReduce has its inherent weakness that it cannot directly support iterations in an algorithm, our approach is a win-win method, since it not only overcomes this weakness, but also secures good accuracy performance. The comparison between this approach and a contemporary algorithm AdaBoost.PL is also performed.

Data streams are large sequences of data, possibly endless and temporarily ordered, that are common-place in Internet of Things (IoT) applications such as intrusion detection in computer networking, fraud detection in financial institutions, real-time tumor tracking in radiotherapy and social media analysis. Algorithms learning from such streams need to be able to construct near real-time models that continuously adapt to potential changes in patterns, in order to retain high performance throughout the stream. It follows that there are numerous challenges involved in supervised learning (or so-called classification) in such environments. One of the challenges in learning from streams is multi-class imbalance, in which the rates of instances in the different class labels differ substantially. Notably, classification algorithms may become biased towards the classes with more frequent instances, sacrificing the performance of the less frequent or so-called minority classes. Further, minority instances often arrive infrequently and in bursts, making accurate model construction problematic. For example, network intrusion detection systems must be able to distinguish between normal traffic and multiple minority classes corresponding to a variety of different types of attacks. Further, having labels for all instances are often infeasible, since we might have missing or late-arriving labels. For instance, when learning from a stream regarding the task of detecting network intrusions, the true label for all instances might not be available, or it might take time until the label is made available, especially for new types of attacks. In this thesis, we contribute to the advancements of online learning from evolving streams by focusing on the above-mentioned areas of multi-class imbalance and missing labels. First, we introduce a multi-class online ensemble algorithm designed to maintain a balanced performance over all classes. Specifically, our approach samples instances with replacement while dynamically increasing the weights of under-represented classes, in order to produce models that benefit all classes. Our experimental results show that our online ensemble method performs well against multi-class imbalanced data in various datasets. We further continue our study by introducing an approach to dealing with missing labels that utilize both labelled and unlabelled data to increase a model’s performance. That is, our method utilizes labelled data for pseudo-labelling unlabelled instances, allowing the model to perform better in environments where labels are scarce. More specifically, our approach features a meta-reinforcement learning agent, trained on multiple-source streams, that can effectively select the prediction of a K nearest neighbours (K-NN) classifier as the label for unlabelled instances. Extensive experiments on benchmark datasets demonstrate the value and effectiveness of our approach and confirm that our method outperforms state-of-the-art.