<p>Graph diffusion is the process of spreading information from one or few nodes to the rest of the graph through edges. The resulting distribution of the information often implies latent structure of the graph where nodes more densely connected can receive more signal. This makes graph diffusions a powerful tool for local clustering, which is the problem of finding a cluster or community of nodes around a given set of seeds. Most existing literatures on using graph diffusions for local graph clustering are linear diffusions as their dynamics can be fully interpreted through linear systems. They are also referred as eigenvector, spectral, or random walk based methods. While efficient, they often have difficulty capturing the correct boundary of a target label or target cluster. On the contrast, maxflow-mincut based methods that can be thought as 1-norm nonlinear variants of the linear diffusions seek to "improve'' or "refine'' a given cluster and can often capture the boundary correctly. However, there is a lack of literature to adopt them for problems such as community detection, local graph clustering, semi-supervised learning, etc. due to the complexity of their formulation. We addressed these issues by performing extensive numerical experiments to demonstrate the performance of flow-based methods in graphs from various sources. We also developed an efficient LocalGraphClustering Python Package that allows others to easily use these methods in their own problems. While studying these flow-based methods, we find that they cannot grow from small seed set. Although there are hybrid procedures that incorporate ideas from both linear diffusions and flow-based methods, they have many hard to set parameters. To tackle these issues, we propose a simple generalization of the objective function behind linear diffusion and flow-based methods which we call generalized local graph min-cut problem. We further show that by involving p-norm in this cut problem, we can develop a nonlinear diffusion procedure that can find local clusters from small seed set and capture the correct boundary simultaneously. Our method can be thought as a nonlinear generalization of the Anderson-Chung-Lang push procedure to approximate a personalized PageRank vector efficiently and is a strongly local algorithm-one whose runtime depends on the size of the output rather than the size of the graph. We also show that the p-norm cut functions improve on the standard Cheeger inequalities for linear diffusion methods. We further extend our generalized local graph min-cut problem and the corresponding diffusion solver to hypergraph-based machine learning problems. Although many methods for local graph clustering exist, there are relatively few for localized clustering in hypergraphs. Moreover, those that exist often lack flexibility to model a general class of hypergraph cut functions or cannot scale to large problems. Our new hypergraph diffusion method on the other hand enables us to compute with a wide variety of cardinality-based hypergraph cut functions and still maintains the strongly local property. We also show that the clusters found by solving the new objective function satisfy a Cheeger-like quality guarantee.</p> <p>Besides clustering, recent work on graph-based learning often focuses on node embeddings and graph neural networks. Although these GNN based methods can beat traditional ones especially when node attributes data is available, it is challenging to understand them because they are highly over-parameterized. To solve this issue, we propose a novel framework that combines topological data analysis and diffusion to transform the complex prediction space into human understandable pictures. The method can be applied to other datasets not in graph formats and scales up to large datasets across different domains and enable us to find many useful insights about the data and the model.</p>

In many relevant scenarios, there is an imbalance between abundant unlabeled data and scarce labeled data to train predictive models. Semi-Supervised Learning and Active Learning are two distinct approaches to deal with this issue. The first one directly uses the unlabeled data to improve model parameter learning, while the second performs a smart choice of unlabeled points to be sent to an annotator, or oracle, which can label these points and increase the labeled training set. In this context, Generative Models are highly appropriate, since they internally represent the data generating process, naturally benefiting from data samples independently of the presence of labels. This Thesis proposes Expectation-Maximization with Density-Weighted Entropy, a novel active semi-supervised learning framework tailored towards generative models. The method is theoretically explored and experiments are conducted to evaluate its application to Gaussian Mixture Models and Multinomial Mixture Models. Based on its partial success, several questions are raised and discussed as to identify possible improvements and decide which shortcomings need to be dealt with before the method is considered robust and generally applicable. / I många relevanta scenarier finns det en obalans mellan god tillgång på oannoterad data och sämre tillgång på annoterad data för att träna prediktiva modeller. Semi-Övervakad Inlärning och Aktiv Inlärning är två distinkta metoder för att hantera denna fråga. Den första använder direkt oannoterad data för att förbättra inlärningen av modellparametrar, medan den andra utför ett smart val av oannoterade punkter som ska skickas till en annoterare eller ett orakel, som kan annotera dessa punkter och öka det annoterade träningssetet. I detta sammanhang är Generativa Modeller mycket lämpliga eftersom de internt representerar data-genereringsprocessen och naturligt gynnas av dataexempel oberoende av närvaron av etiketter. Denna Masteruppsats föreslår Expectation-Maximization med Density-Weighted Entropy, en ny aktiv semi-övervakad inlärningsmetod som är skräddarsydd för generativa modeller. Metoden utforskas teoretiskt och experiment genomförs för att utvärdera dess tillämpning på Gaussiska Mixturmodeller och Multinomiala Mixturmodeller. Baserat på dess partiella framgång ställs och diskuteras flera frågor för att identifiera möjliga förbättringar och avgöra vilka brister som måste hanteras innan metoden anses robust och allmänt tillämplig.

In this thesis, the use of unsupervised and semi-supervised machine learning techniques was analyzed as potential tools for anomaly detection in the sensor network that the electrical system in a Scania truck is comprised of. The experimentation was designed to analyse the need for both point and contextual anomaly detection in this setting. For the point anomaly detection the method of Isolation Forest was experimented with and for contextual anomaly detection two different recurrent neural network architectures using Long Short Term Memory units was relied on. One model was simply a many to one regression model trained to predict a certain signal, while the other was an encoder-decoder network trained to reconstruct a sequence. Both models were trained in an semi-supervised manner, i.e. on data that only depicts normal behaviour, which theoretically should lead to a performance drop on abnormal sequences resulting in higher error terms. In both setting the parameters of a Gaussian distribution were estimated using these error terms which allowed for a convenient way of defining a threshold which would decide if the observation would be flagged as anomalous or not. Additional experimentation's using an exponential weighted moving average over a number of past observations to filter the signal was also conducted. The models performance on this particular task was very different but the regression model showed a lot of promise especially when combined with a filtering preprocessing step to reduce the noise in the data. However the model selection will always be governed by the nature the particular task at hand so the other methods might perform better in other settings. / I den här avhandlingen var användningen av oövervakad och halv-övervakad maskininlärning analyserad som ett möjligt verktyg för att upptäcka avvikelser av anomali i det sensornätverk som elektriska systemet en Scanialastbil består av. Experimentet var konstruerat för att analysera behovet av både punkt och kontextuella avvikelser av anomali i denna miljö. För punktavvikelse av anomali var metoden Isolation Forest experimenterad med och för kontextuella avvikelser av anomali användes två arkitekturer av återkommande neurala nätverk. En av modellerna var helt enkelt många-till-en regressionmodell tränad för att förutspå ett visst märke, medan den andre var ett kodare-avkodare nätverk tränat för att rekonstruera en sekvens.Båda modellerna blev tränade på ett halv-övervakat sätt, d.v.s. på data som endast visar normalt beteende, som teoretiskt skulle leda till minskad prestanda på onormala sekvenser som ger ökat antal feltermer. I båda fallen blev parametrarna av en Gaussisk distribution estimerade på grund av dessa feltermer som tillåter ett bekvämt sätt att definera en tröskel som skulle bestämma om iakttagelsen skulle bli flaggad som en anomali eller inte. Ytterligare experiment var genomförda med exponentiellt viktad glidande medelvärde över ett visst antal av tidigare iakttagelser för att filtera märket. Modellernas prestanda på denna uppgift var välidt olika men regressionmodellen lovade mycket, särskilt kombinerad med ett filterat förbehandlingssteg för att minska bruset it datan. Ändå kommer modelldelen alltid styras av uppgiftens natur så att andra metoder skulle kunna ge bättre prestanda i andra miljöer.

Knowledge Graphs are fascinating concepts in machine learning as they can hold usefully structured information in the form of entities and their relations. Despite the valuable applications of such graphs, most knowledge bases remain incomplete. This missing information harms downstream applications such as information retrieval and opens a window for research in statistical relational learning tasks such as node classification and link prediction. This work proposes a deep learning framework based on existing relational convolutional (R-GCN) layers to learn on highly multi-relational data characteristic of realistic knowledge graphs for node property classification tasks. We propose a deep and improved variant, Deep-RGCNs, with dense and residual skip connections between layers. These skip connections are known to be very successful with popular deep CNN-architectures such as ResNet and DenseNet. In our experiments, we investigate and compare the performance of Deep-RGCN with different baselines on multi-relational graph benchmark datasets, AIFB and MUTAG, and show how the deep architecture boosts the performance in the task of node property classification. We also study the training performance of Deep-RGCNs (with N layers) and discuss the gradient vanishing and over-smoothing problems common to deeper GCN architectures.

Machine learning plays a vital role in preventing financial losses within the banking industry, and still, a lot of state of the art and industry-standard approaches within the field neglect rejected customer information and the potential information that they hold to detect similar risk behavior.This thesis explores the possibility of including this information during training and utilizing transactional history through an LSTM to improve the detection of defaults.  The model is structured so an encoder is first trained with or without rejected customers. Virtual distances are then calculated in the embedding space between the accepted customers. These distances are used to create a graph where each node contains an LSTM network, and a GCN passes messages between connected nodes. The model is validated using two datasets, one public Taiwan dataset and one private Swedish one provided through the collaborative company. The Taiwan dataset used 8000 data points with a 50/50 split in labels. The Swedish dataset used 4644 with the same split.  Multiple metrics were used to validate the impact of the rejected customers and the impact of using time-series data instead of static features. For the encoder part, reconstruction error was used to measure the difference in performance. When creating the edges, the homogeny of the neighborhoods and if a node had a majority of the same labeled neighbors as itself were determining factors, and for the classifier, accuracy, f1-score, and confusion matrix were used to compare results. The results of the work show that the impact of rejected customers is minor when it comes to changes in predictive power. Regarding the effects of using time-series information instead of static features, we saw a comparative result to XGBoost on the Taiwan dataset and an improvement in the predictive power on the Swedish dataset. The results also show the importance of a well-defined virtual distance is critical to the classifier's performance.

Chronic pain (CP) is a personal and economic burden that affects more than 30% of the world's population. While being the leading cause of disability, it is complicated to diagnose and manage. The optimal way to treat CP is to identify the pain mechanism or the underlying cause. The substantial overlap of the pain mechanisms (i.e., Nociceptive, Neuropathic, and Nociplastic) usually makes identification unreachable in a clinical setting where finding the dominant mechanism is complicated. Additionally, many specialists regard CP classification as a spectrum or continuum. Despite the importance, a data-driven way to identify co-existing CP mechanisms and quantification is still absent. This work successfully identified the co-existing CP mechanisms within a patient using Unsupervised Learning while quantifying them without the help of diagnosis established by the clinicians. Two different datasets from different cohorts comprised of patient-reported history and questionnaires were used in this work. Unsupervised Learning (k-prototypes) revealed notable overlaps in the data. It was further emphasized by the outcomes of the Semi-supervised Learning algorithms when the same trend was observed with some diagnosis or class information. It became evident that the CP mechanisms overlap and cannot be classified as distinct conditions. Additionally, mixed pain mechanisms do not make an individual cluster or class, and CP should be considered as a continuum. To reduce data dimension and extract hidden features, Autoencoder was used. Using an overlapping clustering technique, the pain mechanisms were identified. The pain mechanisms were also quantified while elucidating overlaps, and the dominant CP mechanism was successfully pointed out with explainable element. The hamming loss of 0.43 and average precision of 0.5 were achieved when considered as a multi-label classification problem. This work is a data-driven validation that there are significant overlaps in CP conditions, and CP should be considered a continuum where all CP mechanisms may co-exist. / Thesis / Master of Applied Science (MASc) / Chronic pain (CP) is a global burden and the primary cause for patients to seek medical attention. Despite continuous efforts in this area, CP remains clinically challenging to manage. The most effective method of treating CP is identifying the underlying cause or mechanism, which is often unattainable. This thesis attempted to identify the CP mechanisms existing in a patient while quantifying them from patient-reported history and questionnaire data. Unsupervised Learning was used to identify clinically meaningful clusters that revealed the three main CP mechanisms, i.e., Nociceptive, Neuropathic, and Nociplastic, achieving acceptable hamming loss (0.43) and average precision (0.5). The results exhibited that the CP mechanisms co-exist and CP should be regarded as a continuum rather than distinct entities. The algorithm successfully indicated the dominant CP mechanism, a goal for optimal CP management and treatment. The results were also validated by a comparative analysis with data from another cohort that demonstrated a similar trend.

With the development of deep learning methods the requirement of having access to large amounts of data has increased. In this study, we have looked at methods for leveraging unlabeled data while only having access to small amounts of labeled data, which is common in real-world scenarios. We have investigated a method called self-training for leveraging the unlabeled data when training a model. It works by training a teacher model on the labeled data that then labels the unlabeled data for a student model to train on. A popular method in machine learning is ensembling which is a way of improving a single model by combining multiple models. With previous studies mainly focusing on self-training with image data and showing that ensembles can successfully be used for images, we wanted to see if the same applies to text data. We mainly focused on investigating how ensembles can be used as teachers for training a single student model. This was done by creating different ensemble models and comparing them against the individual members in the ensemble. The results showed that ensemble do not necessarily improves the accuracy of the student model over a single model but in certain cases when used correctly they can provide benefits. We found that depending on the dataset bagging BERT models can perform the same or better than a larger BERT model and this translates to the student model. Bagging multiple smaller models also has the benefit of being easier to scale and more computationally efficient to train in comparison to scaling a single model. / Med utvecklingen av metoder för djupinlärning har kravet på att ha tillgång till stora mängder data ökat som är vanligt i verkliga scenarier. I den här studien har vi tittat på metoder för att utnytja oannoterad data när vi bara har tillgång till små mängder annoterad data. Vi har undersökte en metod som kallas självträning för att utnytja oannoterd data när man tränar en modell. Det fungerar genom att man tränar en lärarmodell på annoterad data som sedan annoterar den oannoterade datan för en elevmodell att träna på. En populär metod inom maskininlärning är ensembling som är en teknik för att förbättra en ensam modell genom att kombinera flera modeller. Tidigare studier har främst inriktade på självträning med bilddata och visat att ensembler framgångsrikt kan användas för bild data, vill vi se om detsamma gäller för textdata. Vi fokuserade främst på att undersöka hur ensembler kan användas som lärare för att träna en enskild elevmodell. Detta gjordes genom att skapa olika ensemblemodeller och jämföra dem med de enskilda medlemmarna i ensemblen. Resultaten visade att ensembler inte nödvändigtvis förbättrar elevmodellens noggrannhet jämfört med en enda modell, men i vissa fall kan de ge fördelar när de används på rätt sätt. Vi fann att beroende på datasetet kan bagging av BERT-modeller prestera likvärdigt eller bättre än en större BERT-modell och detta översätts även till studentmodellen prestandard. Att använda bagging av flera mindre modeller har också fördelen av att de är lättare att skala up och mer beräkningseffektivt att träna i jämförelse med att skala up en enskild modell.

<p dir="ltr">The parallel growth of contemporary machine learning (ML) technologies alongside edge/-fog networking has necessitated the development of novel paradigms to effectively manage their intersection. Specifically, the proliferation of edge devices equipped with data generation and ML model training capabilities has given rise to an alternative paradigm called federated learning (FL), moving away from traditional centralized ML common in cloud-based networks. FL involves training ML models directly on edge devices where data are generated.</p><p dir="ltr">A fundamental challenge of FL lies in the extensive heterogeneity inherent to edge/fog networks, which manifests in various forms such as (i) statistical heterogeneity: edge devices have distinct underlying data distributions, (ii) structural heterogeneity: edge devices have diverse physical hardware, (iii) data quality heterogeneity: edge devices have varying ratios of labeled and unlabeled data, and (iv) adversarial compromise: some edge devices may be compromised by adversarial attacks. This dissertation endeavors to capture and model these intricate relationships at the intersection of FL and highly heterogeneous edge/fog networks. To do so, this dissertation will initially develop closed-form expressions for the trade-offs between ML performance and resource cost considerations within edge/fog networks. Subsequently, it optimizes the fundamental processes of FL, encompassing aspects such as batch size control for stochastic gradient descent (SGD) and sampling for global aggregations. This optimization is jointly formulated with networking considerations, which include communication resource consumption and device-to-device (D2D) cooperation.</p><p dir="ltr">In the former half of the dissertation, the emphasis is first on optimizing device sampling for global aggregations in FL, and then on developing a self-sufficient hierarchical meta-learning approach for FL. These methodologies maximize expected ML model performance while addressing common challenges associated with statistical and system heterogeneity. Novel techniques, such as management of D2D data offloading, adaptive CPU clock cycle control, integration of meta-learning, and much more, enable these methodologies. In particular, the proposed hierarchical meta-learning approach enables rapid integration of new devices in large-scale edge/fog networks.</p><p dir="ltr">The latter half of the dissertation directs its ocus towards emerging forms of heterogeneity in FL scenarios, namely (i) heterogeneity in quantity and quality of local labeled and unlabeled data at edge devices and (ii) heterogeneity in terms of adversarially comprised edge devices. To deal with heterogeneous labeled/unlabeled data across edge networks, this dissertation proposes a novel methodology that enables multi-source to multi-target federated domain adaptation. This proposed methodology views edge devices as sources – devices with mostly labeled data that perform ML model training, or targets - devices with mostly unlabeled data that rely on sources’ ML models, and subsequently optimizes the network relationships. In the final chapter, a novel methodology to improve FL robustness is developed in part by viewing adversarial attacks on FL as a form of heterogeneity.</p>

Capacitor discharge welding is an efficient, cost-effective and stable process. It is mostly used for projection welding. Real-time monitoring is desired to ensure quality. Until this point, measured process quantities were evaluated through expert systems. This method takes much time for developing, is strongly restricted to specific welding tasks and needs deep understanding of the process. Another possibility is quality prediction based on process data with machine learning. This method can overcome the downsides of expert systems. But it requires classified welding experiments to achieve a high prediction probability. In industrial manufacturing, it is rarely possible to generate big sets of this type of data. Therefore, semi-supervised learning will be investigated to enable model development on small data sets. Supervised learning is used to develop machine learning models on large amounts of data. These models are used as a comparison to the semi-supervised models. The time signals of the process parameters are evaluated in these investigations. A total of 389 classified weld tests were performed. With semi-supervised learning methods, the amount of training data necessary was reduced to 31 classified data sets.

Text data is easy to retrieve but often expensive to classify, which is why labeled textual data is a resource often lacking in quantity. However, the use of labeled data is crucial in supervised tasks such as text classification, but semi-supervised learning algorithms have shown that the use of unlabeled data during training has the potential to improve model performance, even in comparison to a fully supervised setting. One approach to do semi-supervised learning is consistency training, in which the difference between the prediction distribution of an original unlabeled example and its augmented version is minimized. This thesis explores the performance difference between two techniques for augmenting unlabeled data used for detecting sentiment in movie reviews. The study examines whether the use of augmented data through neural style transfer paraphrasing could achieve comparable or better performance than the use of data augmented through back-translation. Five writing styles were used to generate the augmented datasets: Conversational Speech, Romantic Poetry, Shakespeare, Tweets and Bible. The results show that applying neural style transfer paraphrasing as a data augmentation technique for unlabeled examples in a semi-supervised setting does not improve the performance for sentiment classification with any of the styles used in the study. However, the use of style transferred augmented data in the semi-supervised approach generally performs better than using a model trained in a supervised scenario, where orders of magnitude more labeled data are needed and no augmentation is conducted. The study reveals that the experimented semi-supervised approach is superior to the fully supervised setting but worse than the semi-supervised approach using back-translation. / Textdata är lätt att få tag på men dyr att beteckna, vilket är varför annoterad textdata ofta inte finns i stora kvantiteter. Annoterad data är dock av yttersta vikt för övervakad inlärning, exempelvis för textklassificering, men semiövervakade inlärningsalgoritmer har visat att användandet av textdata utan annoteringar har potential att förbättra en inlärningsalgoritms resultat, även i jämförelse med helt övervakade algoritmer. Ett semi-övervakad inlärningsteknik är konsistensträning, där skillnaden mellan inferensen på en oförändrad datapunkt och en förändrar datapunkt minimeras. Denna uppsats utforskar skillnaden i resultat av att använda två olika tekniker för att förändra data som inte är annoterad för att detektera sentiment i filmrecensioner. Studien undersöker huruvida data förändrad via neural stilöverföring kan åstadkomma jämförbara eller bättre resultat i jämförelse med data förändrad genom tillbaka-översättning. Fem olika skrivstilar använda för att generera den förändrade datan: konversationellt tal, romantisk poesi, Shakespeare, Twitter-skrift samt Bibel. Resultaten visar att applicera neural stilöverföring på att förändra ej annoterade exempel för konsistensträning inte förbättrar resultaten i jämförelse med tillbaka-översättning. Semi-övervakad inlärning med stiltransferering presterar dock generellt bättre än en fullt övervakad, jämbördig algoritm som behöver flera magnituder fler annoteringar. Studien visar att den semiövervakade inlärningstekniken är bättre än den fullt övervakade modellen, men sämre än den semi-övervakade tekniken som använder tillbaka-översättning.