Multi-Task Learning (MTL) has been an active research area in machine learning for two decades. By training multiple relevant tasks simultaneously with information shared across tasks, it is possible to improve the generalization performance of each task, compared to training each individual task independently. During the past decade, most MTL research has been based on the Regularization-Loss framework due to its flexibility in specifying various types of information sharing strategies, the opportunity it offers to yield a kernel-based methods and its capability in promoting sparse feature representations. However, certain limitations exist in both theoretical and practical aspects of Regularization-Loss-based MTL. Theoretically, previous research on generalization bounds in connection to MTL Hypothesis Space (HS)s, where data of all tasks are pre-processed by a (partially) common operator, has been limited in two aspects: First, all previous works assumed linearity of the operator, therefore completely excluding kernel-based MTL HSs, for which the operator is potentially non-linear. Secondly, all previous works, rather unnecessarily, assumed that all the task weights to be constrained within norm-balls, whose radii are equal. The requirement of equal radii leads to significant inflexibility of the relevant HSs, which may cause the generalization performance of the corresponding MTL models to deteriorate. Practically, various algorithms have been developed for kernel-based MTL models, due to different characteristics of the formulations. Most of these algorithms are a burden to develop and end up being quite sophisticated, so that practitioners may face a hard task in interpreting and implementing them, especially when multiple models are involved. This is even more so, when Multi-Task Multiple Kernel Learning (MT-MKL) models are considered. This research largely resolves the above limitations. Theoretically, a pair of new kernel-based HSs are proposed: one for single-kernel MTL, and another one for MT-MKL. Unlike previous works, we allow each task weight to be constrained within a norm-ball, whose radius is learned during training. By deriving and analyzing the generalization bounds of these two HSs, we show that, indeed, such a flexibility leads to much tighter generalization bounds, which often results to significantly better generalization performance. Based on this observation, a pair of new models is developed, one for each case: single-kernel MTL, and another one for MT-MKL. From a practical perspective, we propose a general MT-MKL framework that covers most of the prominent MT-MKL approaches, including our new MT-MKL formulation. Then, a general purpose algorithm is developed to solve the framework, which can also be employed for training all other models subsumed by this framework. A series of experiments is conducted to assess the merits of the proposed mode when trained by the new algorithm. Certain properties of our HSs and formulations are demonstrated, and the advantage of our model in terms of classification accuracy is shown via these experiments.

Generalization error describes how well a supervised machine learning algorithm predicts the labels of input data that it has not been trained with. This project aims to explore two different methods for bounding generalization error, f-CMI and ISMI, which explicitly use mutual information. Our experiments are based on the experiments in the papers in which the methods were proposed. The experiments implement and validate the accuracy of the mathematically derived bounds. Each methodology also has a different method for calculating mutual information. The ISMI bound experiment used a multivariate normal distribution dataset, whereas a dataset consisting of cats and dogs was used for the experiment using f-CMI. Our results show that both methods are capable of bounding the generalization error of a binary classification algorithm and provide bounds that closely follow the true generalization error. The results of the experiments agree with the original experiments, indicating that the proposed methods also work for similar applications with different datasets. / Generaliseringsfel beskriver hur väl en övervakad maskininlärnings algoritm förutspår etiketter av indata som den inte har blivit tränad med. Syftet med projektet är att utforska två olika metoder för att begränsa generaliseringsfelet, f-CMI och ISMI som explicit använder ömsesidig information. Vårt experiment är baserat på experimenten i artiklarna som tog fram metoderna. Experimenten implementerade och validerade noggrannheten av de matematiskt härleda gränserna. Varje metod har olika sätt att beräkna den ömsesidiga informationen. ISMI gräns experimentet använde en flerdimensionell normalfördelning som data set, medan en datauppsättning med katter och hundar användes för f-CMI gränsen. Våra resultat visar att båda metoder kan begränsa generaliseringsfelet av en binär klassificerings algoritm och förse gränser som nära följer det sanna generaliseringsfelet. Resultatet av experimenten instämmer med de ursprungliga författarnas experiment vilket indikerar att de föreslagna metoderna också fungerar for liknande tillämpningar med andra data set. / Kandidatexjobb i elektroteknik 2022, KTH, Stockholm