Neural networks (NN) have demonstrated considerable capabilities in tackling tasks in a diverse set of fields, including natural language processing, image classification, and regression. In recent years, the amount of available data to train Deep Learning (DL) models has increased tremendously, thus requiring larger and larger models to learn the underlying patterns in the data. Inference time, communication cost in the distributed case, required storage resources, and computational capabilities have increased proportional to the model's size, thus making NNs less suitable for two cases: i) tasks requiring low inference time (e.g., real-time monitoring) and ii) training on low powered devices. These two cases, which have become crucial in the last decade due to the pervasiveness of low-powered devices and NN models, are addressed in this licentiate thesis. As the first contribution, we analyze the distributed case with multiple low-powered devices in a federated scenario. Cross-device Federated Learning (FL) is a branch of Machine Learning (ML) where multiple participants train a common global model without sharing data in a centralized location. In this thesis, a novel technique named Coded Federated Dropout (CFD) is proposed to carefully split the global model into sub-models, thus increasing communication efficiency and reducing the burden on the devices with only a slight increase in training time. We showcase our results for an example image classification task. As the second contribution, we consider the anomaly detection task on Electrocardiogram (ECG) recordings and show that including prior knowledge in NNs models drastically reduces model size, inference time, and storage resources for multiple state-of-the-art NNs. In particular, this thesis focuses on AEs, a subclass of NNs, which is suitable for anomaly detection. I propose a novel approach, called FMM-Head, which incorporates basic knowledge of the ECG waveform shape into an AE. The evaluation shows that we improve the AUROC of baseline models while guaranteeing under-100ms inference time, thus enabling real-time monitoring of ECG recordings from hospitalized patients. Finally, several potential future works are presented. The inclusion of prior knowledge can be further exploited in the ECG Imaging (ECGI) case, where hundreds of ECG sensors are used to reconstruct the 3D electrical activity of the heart. For ECGI, the reduction in the number of sensors employed (i.e., the input space) is also beneficial in terms of reducing model size. Moreover, this thesis advocates additional techniques to integrate ECG anomaly detection in a distributed and federated case. / Neurala nätverk (NN) har visat god förmåga att tackla uppgifter inom en mängd olika områden, inklusive Natural Language Processing (NLP), bildklassificering och regression. Under de senaste åren har mängden tillgänglig data för att träna Deep Learning (DL)-modeller ökat enormt, vilket kräver större och större modeller för att lära sig de underliggande mönstren i datan. Inferens tid och kommunikationskostnad i det distribuerade fallet, nödvändiga lagringsresurser och beräkningskapacitet har ökat proportionerligt mot modellens storlek vilket gör NN mindre lämpliga använda i två fall: (i) uppgifter som kräver snabba slutledningar (t.ex. realtidsövervakning) och (ii) användning på mindre kraftfulla enheter. De här två fallen, som har blivit mer förekommande under det senaste decenniet på grund av omfattningen av mindre kraftfulla enheter och NN-modeller, behandlas i denna licentiatuppsats. Som det första bidraget analyserar vi det distribuerade fallet med flera lättdrivna enheter i ett federerat scenario. Cross-device Federated Learning (FL) är en gren av Machine Learning (ML) där flera deltagare tränar en gemensam global modell utan att dela data på en centraliserad plats. I denna avhandling föreslås en nyteknik, Coded Federated Dropout (CFD), som delar upp den globala modellen i undermodeller, vilket ökar kommunikationseffektiviteten och samtidigt minskar belastningen på enheterna. Detta erhålls med endast en liten förlängning av träningstiden. Vi delger våra resultat för en exempeluppgift för bildklassificering. Som det andra bidraget betraktar vi anomalidetekteringsuppgiften Elektrokardiogram (EKG)-registrering och visar att inklusionen av förkunskaper i NN-modeller drastiskt minskar modellstorlek, inferenstider och lagringsresurser för flera moderna NN. Speciellt fokuserar denna avhandling på Autoencoders (AEs), en delmängd av NN, lämplig för avvikelsedetektering. En ny metod, kallad FMM-Head, föreslås. vilken  omformar grundläggande kunskaper om EKG-vågformen till en AE. Utvärderingen visar att vi förbättrar arean under kurvan (AUROC) för baslinjemodeller samtidigt som vi garanterar under 100 ms inferenstid, vilket möjliggör realtidsövervakning av EKG-inspelningar från inlagda patienter.  Slutligen presenteras flera potentiella framtida utvidgningar. Införandet av tidigare kunskap kan utnyttjas ytterligare i fallet med EKG Imaging (ECGI), där hundratals EKG-sensorer används för att rekonstruera den elektriska 3D-aktiviteten hos hjärtat. För ECGI är minskningen av antalet använda sensorer (dvs inmatningsutrymme) också fördelaktig när det gäller att minska modellstorleken. Dessutom förespråkas i denna avhandling ytterligare tekniker för att integrera EKG-avvikelsedetektering i distribuerade och federerade fall. / <p>This research leading to this thesis is based upon work supported by the King Abdullah University of Science and Technology (KAUST) Office of Research Administration (ORA) under Award No. ORA-CRG2021-4699</p>

The need for a method to create a collaborative machine learning model which can utilize data from different clients, each with privacy constraints, has recently emerged. This is due to privacy restrictions, such as General Data Protection Regulation, together with the fact that machine learning models in general needs large size data to perform well. Google introduced federated learning in 2016 with the aim to address this problem. Federated learning can further be divided into horizontal and vertical federated learning, depending on how the data is structured at the different clients. Vertical federated learning is applicable when many different features is obtained on distributed computation nodes, where they can not be shared in between. The aim of this thesis is to identify the current state of the art methods in vertical federated learning, implement the most interesting ones and compare the results in order to draw conclusions of the benefits and drawbacks of the different methods. From the results of the experiments, a method called FedBCD shows very promising results where it achieves massive improvements in the number of communication rounds needed for convergence, at the cost of more computations at the clients. A comparison between synchronous and asynchronous approaches shows slightly better results for the synchronous approach in scenarios with no delay. Delay refers to slower performance in one of the workers, either due to lower computational resources or due to communication issues. In scenarios where an artificial delay is implemented, the asynchronous approach shows superior results due to its ability to continue training in the case of delays in one or several of the clients.

För varje dag som går möter stora industrier en ökad mängd intrång i sina IT-system. De flesta befintliga verktyg som använder sig utav maskininlärning är starkt beroende av stora mängder data, vilket innebär risker under dataöverföringen. Därför har syftet med denna studie varit att undersöka om en decentraliserad integritetsbevarande strategi kan vara ett bra alternativ för att minska effektiviteten av dessa attacker. Mer specifikt skulle användningen av Random Forests, en av de mest populära algoritmerna för maskininlärning, kunna utökas med decentraliseringstekniken Federated Learning tisammans med Differential Privacy, för att skapa en ideal metod för att upptäcka nätverksintrång? Med hjälp av befintliga kodbibliotek för maskininlärnings och verklighetsbaserad data har detta projekt konstruerat olika modeller för att simulera hur väl olika decentraliserade och integritetsbevarande modeller kan jämföras med traditionella alternativ. De skapade modellerna innehåller antingen Federated Learning, Differential Privacy eller en kombination av båda. Huvuduppgiften för dessa modeller är att förbättra integriteten och samtidigt minimera minskningen av precision. Resultaten indikerar att båda teknikerna kommer med en liten minskning i noggrannhet jämfört med traditionella alternativ. Huruvida precisionsförlusten är acceptabel eller beror på det specifika användningsområdet. Det utvecklade kombinerade alternativet lyckades dock inte nå acceptabel precision vilket hindrar oss från att dra några slutsatser. / With each passing day, large industries face an increasing amount of intrusions into their IT environments. Most existing machine learning countermeasures heavily rely on large amounts of data which introduces risk during the data-transmission. Therefore, the objective of this study has been to investigate whether a decentralized privacy-preserving approach could be a sensible alternative to decrease the effectiveness of these attacks. More specifically could the use of Random Forests, one of the most popular machine learning algorithms, be extended using the decentralization technique Federated Learning in cooperation with Differential Privacy, in order to create an ideal approach for network intrusion detection? With the assistance of existing machine learning code-libraries and real-life data, this thesis has constructed various experimental models to simulates how well different decentralized and privacy-preserving approaches compare to traditional ones. The models created incorporate either Federated Learning, Differential Privacy or a combination of both. The main task of these models is to enhance privacy while minimizing the decrease in accuracy. The results indicate that both techniques comes with a small decrease in accuracy compared to traditional alternatives. whether the accuracy loss is acceptable or not may depend on the specific scenario. The developed combined approach however, failed to reach acceptable accuracy which prevents us from drawing any conclusions.

No description available.

Background: Human-computer interaction (HCI) is one of the daily triggering emotional events in today’s world and researchers in this area have been exploring different techniques to enhance emotional ability in computers. Due to privacy concerns and the laboratory's limited capability for gathering data from a large number of users, common machine learning techniques that are extensively used in emotion recognition tasks lack adequate data collection. To address these issues, we propose a decentralized framework based on the Federated Learning architecture where raw data is collected and analyzed locally. The effects of these analyses in large numbers of updates are transferred to a server to aggregate for the creation of a global model for the emotion recognition task using only Galvanic Skin Response (GSR) signals and their extracted features.  Objectives: This thesis aims to explore how the CNN based federated learning approach can be used in emotion recognition considering data privacy protection and investigate if it reaches the same performance as basic centralized CNN.Methods: To investigate the effect of the proposed method in emotion recognition, two architectures including centralized and federated are designed with the CNN model. Then the results of these two architectures are compared to each other. The dataset used in our work is the CASE dataset. In federated architecture, we employ neurons and weights to train the models instead of raw data, which is used in the centralized architecture.  Results: The performance results indicate that the proposed model not only can work well but also performs better than some other related work methods regarding valance accuracy. Besides, it also has the ability to collect more data from various sources and also protecting sensitive users’ data better by supporting tighter privacy regulations. The physiological data is inherently anonymous but when it comes to using it with other modalities such as video or voice, maintaining the same anonymity is challenging.  Conclusions: This thesis concludes that the federated CNN based model can be used in emotion recognition systems and obtains the same accuracy performance as centralized architecture. Regarding classifying the valance, it outperforms some other state-of-the-art methods. Meanwhile, its federated nature can provide better privacy protection and data diversity for the emotion recognition system.

<div>Federated learning (FL) is a decentralized privacy-preserving learning technique in which clients learn a joint collaborative model through a central aggregator without sharing their data. In this setting, all clients learn a single common predictor (FedAvg), which does not generalize well on each client's local data due to the statistical data heterogeneity among clients. In this paper, we address this problem with PersFL, a discrete two-stage personalized learning algorithm. In the first stage, PersFL finds the optimal teacher model of each client during the FL training phase. In the second stage, PersFL distills the useful knowledge from optimal teachers into each user's local model. The teacher model provides each client with some rich, high-level representation that a client can easily adapt to its local model, which overcomes the statistical heterogeneity present at different clients. We evaluate PersFL on CIFAR-10 and MNIST datasets using three data-splitting strategies to control the diversity between clients' data distributions.</div><div><br></div><div>We empirically show that PersFL outperforms FedAvg and three state-of-the-art personalization methods, pFedMe, Per-FedAvg and FedPer on majority data-splits with minimal communication cost. Further, we study the performance of PersFL on different distillation objectives, how this performance is affected by the equitable notion of fairness among clients, and the number of required communication rounds. We also build an evaluation framework with the following modules: Data Generator, Federated Model Generation, and Evaluation Metrics. We introduce new metrics for the domain of personalized FL, and split these metrics into two perspectives: Performance, and Fairness. We analyze the performance of all the personalized algorithms by applying these metrics to answer the following questions: Which personalization algorithm performs the best in terms of accuracy across all the users?, and Which personalization algorithm is the fairest amongst all of them? Finally, we make the code for this work available at https://tinyurl.com/1hp9ywfa for public use and validation.</div>

Due to privacy and regulatory reasons, sharing data between institutions can be difficult. Because of this, real-world data are not fully exploited by machine learning (ML). An emerging method is to train ML models with federated learning (FL) which enables clients to collaboratively train ML models without sharing raw training data. We explored peer-to-peer FL by extending a prominent centralized FL algorithm called Fedavg to function in a peer-to-peer setting. We named this extended algorithm FedavgP2P. Deep neural networks at 100 simulated clients were trained to recognize digits using FedavgP2P and the MNIST data set. Scenarios with IID and non-IID client data were studied. We compared FedavgP2P to Fedavg with respect to models' convergence behaviors and communication costs. Additionally, we analyzed the connection between local client computation, the number of neighbors each client communicates with, and how that affects performance. We also attempted to improve the FedavgP2P algorithm with heuristics based on client identities and per-class F1-scores. The findings showed that by using FedavgP2P, the mean model convergence behavior was comparable to a model trained with Fedavg. However, this came with a varying degree of variation in the 100 models' convergence behaviors and much greater communications costs (at least 14.9x more communication with FedavgP2P). By increasing the amount of local computation up to a certain level, communication costs could be saved. When the number of neighbors a client communicated with increased, it led to a lower variation of the models' convergence behaviors. The FedavgP2P heuristics did not show improved performance. In conclusion, the overall findings indicate that peer-to-peer FL is a promising approach.

<p>We consider the problem of developing innovative machine learning tools for online education and evaluate their ability to provide instructional resources.  Prediction tasks for student behavior are a complex problem spanning a wide range of topics: we complement current research in student grade prediction and clickstream analysis by considering data from three areas of online learning: Social Learning Networks (SLN), Instructor Feedback, and Learning Management Systems (LMS). In each of these categories, we propose a novel method for modelling data and an associated tool that may be used to assist students and instructors. First, we develop a methodology for analyzing instructor-provided feedback and determining how it correlates with changes in student grades using NLP and NER--based feature extraction. We demonstrate that student grade improvement can be well approximated by a multivariate linear model with average fits across course sections approaching 83\%, and determine several contributors to student success. Additionally, we develop a series of link prediction methodologies that utilize spatial and time-evolving network architectures to pass network state between space and time periods. Through evaluation on six real-world datasets, we find that our method obtains substantial improvements over Bayesian models, linear classifiers, and an unsupervised baseline, with AUCs typically above 0.75 and reaching 0.99. Motivated by Federated Learning, we extend our model of student discussion forums to model an entire classroom as a SLN. We develop a methodology to represent student actions across different course materials in a shared, low-dimensional space that allows characteristics from actions of different types to be passed jointly to a downstream task. Performance comparisons against several baselines in centralized, federated, and personalized learning demonstrate that our model offers more distinctive representations of students in a low-dimensional space, which in turn results in improved accuracy on a common downstream prediction task. Results from these three research thrusts indicate the ability of machine learning methods to accurately model student behavior across multiple data types and suggest their ability to benefit students and instructors alike through future development of assistive tools. </p>

There is a potential in the field of medicine and finance of doing collaborative machine learning. These areas gather data which can be used for developing machine learning models that could predict all from sickness in patients to acts of economical crime like fraud. The problem that exists is that the data collected is mostly of confidential nature and should be handled with precaution. This makes the standard way of doing machine learning - gather data at one centralized server - unwanted to achieve. The safety of the data have to be taken into account. In this project we will explore the Federated learning approach of ”bringing the code to the data, instead of data to the code”. It is a decentralized way of doing machine learning where models are trained on connected devices and data is never shared. Keeping the data privacypreserved.

No description available.