<p>This dissertation explore two major goals in Deep Learning algorithm design: efficiency and trustworthiness. We motivate these concerns in Chapter 1 and give relevant background in Chapter 2. We then discuss six works to target these two goals. </p> <p>The first of these discusses how to make the model compression methodology more efficient, so it can be done in a single shot. This allows us to create models with reduced size and layers, so we can have faster and more efficient inference, and is covered in Chapter 3. We then extend this to target efficiency in continual learning in Chapter 4, while mitigating the problem of catastrophic forgetting. The method discussed also allows us to circumvent the potential for data leakage by avoiding the need to store any data from the past tasks. Next, we consider brain-inspired computing as an alternative to traditional neural networks to improve compute efficiency of networks. The spiking neural networks discussed however have large inference latency due to the need for accumulating spikes over many timesteps. We tackle this by introducing a new scheme that distributes an image over time by breaking it down into a sum of its ranked sinusoidal bases in Chapter 5. This results in networks that are faster and more efficient to deploy. Chapter 6 targets mitigating both the communication expense and potential for data leakage in federated learning, by distilling the gradients to be communicated in a small number of images that resemble noise. Communicating these images is more efficient, and circumvents the potential for data leakage as they resemble noise. We then explore the applications of studying curvature of loss with respect to input data points in the last two chapters. We first utilize curvature to create performant coresets to reduce the size of datasets, to make training more efficient in Chapter 7. In Chapter 8, we use curvature as a metric for overfitting and use it to expose dataset integrity issues arising from memorization.</p>

Natural Language Processing (NLP) is one of the most popular and visible forms of Artificial Intelligence in recent years. This is partly because it has to do with a common characteristic of human beings: language. NLP applications allow to create new services in the industrial sector in order to offer new solutions and provide significant productivity gains. All of this has happened thanks to the rapid progression of Deep Learning models. Large scale contextual representation models, such asWord2Vec, ELMo and BERT, have significantly advanced NLP in recently years. With these latest NLP models, it is possible to understand the semantics of text to a degree never seen before. However, they require large amounts of text data to process to achieve high-quality results. This data can be gathered from different sources, but one of the main collection points are devices such as smartphones, smart appliances and smart sensors. Lamentably, joining and accessing all this data from multiple sources is extremely challenging due to privacy and regulatory reasons. New protocols and techniques have been developed to solve this limitation by training models in a massively distributed manner taking advantage of the powerful characteristic of the devices that generates the data. Particularly, this research aims to test the viability of training NLP models, in specific Word2Vec, with a massively distributed protocol like Federated Learning. The results show that FederatedWord2Vecworks as good as Word2Vec is most of the scenarios, even surpassing it in some semantics benchmark tasks. It is a novel area of research, where few studies have been conducted, with a large knowledge gap to fill in future researches. / Naturlig språkbehandling är en av de mest populära och synliga formerna av artificiell intelligens under de senaste åren. Det beror delvis på att det har att göra med en gemensam egenskap hos människor: språk. Naturlig språkbehandling applikationer gör det möjligt att skapa nya tjänster inom industrisektorn för att erbjuda nya lösningar och ge betydande produktivitetsvinster. Allt detta har hänt tack vare den snabba utvecklingen av modeller för djup inlärning. Modeller i storskaligt sammanhang, som Word2Vec, ELMo och BERT har väsentligt avancerat naturligt språkbehandling på senare tid år. Med dessa senaste naturliga språkbearbetningsmo modeller är det möjligt att förstå textens semantik i en grad som aldrig sett förut. De kräver dock stora mängder textdata för att bearbeta för att uppnå högkvalitativa resultat. Denna information kan samlas in från olika källor, men ett av de viktigaste insamlingsställena är enheter som smartphones, smarta apparater och smarta sensorer. Beklagligtvis är det extremt utmanande att gå med och komma åt alla dessa uppgifter från flera källor på grund av integritetsskäl och regleringsskäl. Nya protokoll och tekniker har utvecklats för att lösa denna begränsning genom att träna modeller på ett massivt distribuerat sätt med fördel av de kraftfulla egenskaperna hos enheterna som genererar data. Särskilt syftar denna forskning till att testa livskraften för att utbilda naturligt språkbehandling modeller, i specifika Word2Vec, med ett massivt distribuerat protokoll som Förenat Lärande. Resultaten visar att det Förenade Word2Vec fungerar lika bra som Word2Vec är de flesta av scenarierna, till och med överträffar det i vissa semantiska riktmärken. Det är ett nytt forskningsområde, där få studier har genomförts, med ett stort kunskapsgap för att fylla i framtida forskningar.

This study explored how clients with distorted data affected the Federated Learning process using the FedAvg and FedProx algorithms. Different amounts of the three distortions, Translation, Rotation, and Blur, were tested using three different Machine Learning models. The models were a Dense network, the well-known convolutional network LeNet-5, and a smaller version of the ResNet architecture. The results of the study successfully showcases how different distortions affect the three models. Therefore, they also show that the risk of local data distortion is important to factor in when picking a Machine Learning model for Federated Learning.

Federated learning is a widely established modern machine learning methodology where training is done directly on the client device with local client data and the local training results are shared to compute a global model. Federated learning emerged as a result of data ownership and the privacy concerns of traditional machine learning methodologies where data is collected and trained at a central location. However, in a distributed data environment, the training suffers significantly when the client data is not identically distributed. Hence, clustered federated learning was proposed where similar clients are clustered and trained independently to form specialized cluster models which are then used to compute a global model. In this approach, the cluster selection for clustered federated learning is a major factor that affects the effectiveness of the global model. This research presents two approaches for client clustering using local client data for clustered federated learning while preserving data privacy. The two proposed approaches use min-wise independent permutations to compute client signatures using text and word embeddings. These client signatures are then used as a representation of client data to cluster clients using agglomerative hierarchical clustering. Unlike previously proposed clustering methods, the two presented approaches do not use model updates, provide a better privacy-preserving mechanism and have a lower communication overhead. With extensive experimentation, we show that the proposed approaches outperform the random clustering approach. Finally, we present a client clustering methodology that can be utilized in a practical clustered federated learning environment. / Federerad inlärning är en etablerad och modern maskininlärnings metod. Träningen är utförd direkt på klientenheten med lokal klient data. Sen är dem lokala träningsresultat delad för att beräkna en global modell. Federerad inlärning har utvecklats på grund av dataägarskap- och dataintegritetsproblem vid traditionella maskininlärnings metoder. Dessa metoder samlar och tränar data på en central enhet. I den här metoden är kluster selektionen en viktig faktor som påverkar effektiviteten av den globala modellen. Detta forskningsarbete presenterar två metoder för klient klustring med hjälp av lokala klientdata för federerad inlärning samtidigt tar metoderna hänsyn på dataintegritet. Metoderna använder “min-wise” oberoende permutations och förtränade (“text och word”) inbäddningar. Dessa klientsignaturer används som en klientdata representation för att klustrar klienter med hjälp av agglomerativ hierarkisk klustring. Till skillnad från tidigare klustringsmetoder använder de två presenterade metoderna inte modelluppdateringar. Detta ger en bättre sekretessbevarande mekanism och har lägre kommunikationskostnader. De två presenterade metoderna överträffar den slumpmässiga klustringsmetoden genom omfattande experiment och analys. Till slut presenterar vi en klientklustermetodik som kan användas i en praktisk klustrad federerad inlärningsmiljö.

Federated Learning now a days has emerged as a promising standard for machine learning model training, which can be executed collaboratively on decentralized data sources. As the adoption of Federated Learning grows, the selection of the apt frame work for our use case has become more important. This study mainly concentrates on a comprehensive overview of three prominent Federated Learning frameworks Flower, FedN, and FedML. The performance, scalability, and utilization these frame works is assessed on the basis of an NLP use case. The study commences with an overview of Federated Learning and its significance in distributed learning scenarios. Later on, we explore into the examination of the Flower framework in-depth covering its structure, communication methods and interaction with deep learning libraries. The performance of Flower is evaluated by conducting experiments on a standard benchmark dataset. Metrics provide measurements for accuracy, speed and scalability. Tests are also conducted to assess Flower's ability to handle large-scale Federated Learning setups. The same is carried out with the other two frameworks FedN and FedML also. To gain better insight into the strengths, limitations, and suitability of Flower, FedN, and FedML for different Federated Learning scenarios, the study utilizes the above stated comparative analysis on a real time use case. The possibilities for integrating these frameworks with current machine learning workflows are discussed. Furthermore, the final results and conclusions may help researchers and practitioners to make conversant decisions regarding framework selection for their Federated Learning applications. / <p>Det finns övrigt digitalt material (t.ex. film-, bild- eller ljudfiler) eller modeller/artefakter tillhörande examensarbetet som ska skickas till arkivet.</p><p>There are other digital material (eg film, image or audio files) or models/artifacts that belongs to the thesis and need to be archived.</p>

In domains where data is sensitive or private, there is a great value in methods that can learn in a distributed manner without the data ever leaving the local devices. Federated Learning (FL) has recently emerged as a promising solution to collaborative machine learning challenges while maintaining data privacy. With FL, multiple entities, whether cross-device or cross-silo, can jointly train models without compromising the locality or privacy of their data. Instead of moving data to a central storage system or cloud for model training, code is moved to the data owners’ local sites, and incremental local updates are combined into a global model. In this way FL enhances data pri-vacy and reduces the probability of eavesdropping to a certain extent. In this thesis we have utilized the means of Federated Learning into a Real-Time Object Detection (RTOB) model in order to investigate its performance and privacy awareness towards a traditional centralized ML environment. Several object detection models have been built us-ing YOLO framework and training with a custom dataset for indoor object detection. Local tests have been performed and the most opti-mal model has been chosen by evaluating training and testing metrics and afterwards using NVIDIA Jetson Nano external device to train the model and integrate into a Federated Learning environment using an open-source FL framework. Experiments has been conducted through the path in order to choose the optimal YOLO model (YOLOv8) and the best fitted FL framework to our study (FEDn).We observed a gradual enhancement in balancing the APC factors (Accuracy-Privacy-Communication) as we transitioned from basic lo-cal models to the YOLOv8 implementation within the FEDn system, both locally and on the SSC Cloud production environment. Although we encountered technical challenges deploying the YOLOv8-FEDn system on the SSC Cloud, preventing it from reaching a finalized state, our preliminary findings indicate its potential as a robust foundation for FL applications in RTOB models at the edge.

In recent years, companies of all sizes have become increasingly dependent on customer user data and processing it using machine learning (ML) methods. These methods do, however, require the raw user data to be stored locally on a server or cloud service, raising privacy concerns. Hence, the purpose of this paper is to analyze a new alternative ML method, called federated learning (FL). FL allows the data to remain on each respective device while still being able to create a global model by averaging local models on each client device. The analysis in this report is based on two different types of simulations. The first is simulations in a virtual environment where a larger number of devices can be included, while the second is simulations on a physical testbed of Raspberry Pi (RPI) single-board computers. Different parameters are changed and altered to find the optimal performance, accuracy, and loss of computations in each case. The results of all simulations show that fewer clients and more training epochs increase the accuracy when using independent and identically distributed (IID) data. However, when using non-IID data, the accuracy is not dependent on the number of epochs, and it becomes chaotic when decreasing the number of clients which are sampled each round. Furthermore, the tests on the RPIs show results which agree with the virtual simulation. / På den senaste tiden har företag blivit allt mer beroende av ku rs användardata och har börjat använda maskininlärningsmodeller för att processera datan. För att skapa dessa modeller behövs att användardata lagras lokalt på en server eller en molntjänst, vilket kan leda till integritetsproblematik. Syftet med denna rapport är därför att analysera en ny alternativ metod, vid namn ”federated learning” (FL). Denna metod möjliggör skapandet av en global modell samtidigt som användardata förblir kvar på varje klients enhet. Detta görs genom att den globala modellen bestäms genom att beräkna medelvärdet av samtliga enheters lokala modeller. Analysen av metoden görs baserat på två olika typer av simuleringar. Den första görs i en virtuell miljö för att kunna inkluderastörre mängder klientenheter medan den andra typen görs på en fysisk testbädd som består av enkortsdatorerna Raspberry Pi (RPI). Olika parametrar justeras och ändras för att finna modellens optimala prestanda och nogrannhet. Resultaten av simuleringarna visar att färre klienter och flera träningsepoker ökar noggrannheten när oberoende och likafördelad (på engelska förkortat till IID) data används. Däremot påvisas att noggrannheten inte är beroende av antalet epoker när icke-IID data nyttjas. Noggrannheten blir däremot kaotisk när antalet klienter som används för att träna på varje runda minskas. Utöver observeras det även att testresultaten från RPI enheterna stämmer överens med resultatet från simuleringarna. / Kandidatexjobb i elektroteknik 2022, KTH, Stockholm

Timely data collection and execution in heterogeneous Internet of Things (IoT) networks in which different protocols and spectrum bands coexist such as WiFi, RFID, Zigbee, and LoRa, requires further investigation. This thesis studies the problem of age-of-information minimization in heterogeneous IoT networks consisting of heterogeneous IoT devices, an intermediate layer of multi-protocol mobile gateways (M-MGs) that collects and relays data from IoT objects and performs computing tasks, and heterogeneous access points (APs). A federated matching framework is presented to model the collaboration between different service providers (SPs) to deploy and share M-MGs and minimize the average weighted sum of the age-of-information and energy consumption. Further, we develop a two-level multi-protocol multi-agent actor-critic (MP-MAAC) to solve the optimization problem, where M-MGs and SPs can learn collaborative strategies through their own observations. The M-MGs' strategies include selecting IoT objects for data collection, execution, relaying, and/or offloading to SPs’ access points while SPs decide on spectrum allocation. Finally, to improve the convergence of the learning process we incorporate federated learning into the multi-agent collaborative framework. The numerical results show that our Fed-Match algorithm reduces the AoI by factor four, collects twice more packets than existing approaches, reduces the penalty by factor five when enabling relaying, and establishes design principles for the stability of the training process.

Water monitoring is an essential process that managesthe well-being of freshwater ecosystems. However, it isgenerally an inefficient process as most data collection is donemanually. By combining wireless sensor technology and machinelearning techniques, projects such as iWater aim to modernizecurrent methods. The purpose of the iWater project is to developa network of smart sensors capable of collecting and analyzingwater quality-related data in real time.To contribute to this goal, a comparative study between theperformance of a centralized machine learning algorithm thatis currently used, and a distributed model based on a federatedlearning algorithm was done. The data used for training andtesting both models was collected by a wireless sensor developedby the iWater project. The centralized algorithm was used asthe basis for the developed distributed model. Due to lack ofsensors, the distributed model was simulated by down-samplingand dividing the sensor data into six data sets representing anindividual sensor. The results are similar for both models andthe developed algorithm reaches an accuracy of 98.41 %. / Vattenövervakning är en nödvändig processför att få inblick i sötvattensekosystems välmående. Dessvärreär det en kostsam och tidskrävande process då insamling avdata vanligen görs manuellt. Genom att kombinera trådlössensorteknologi och maskininlärnings algoritmer strävar projektsom iWater mot att modernisera befintliga metoder.Syftet med iWater är att skapa ett nätverk av smarta sensorersom kan samla in och analysera vattenkvalitetsrelaterade datai realtid. För att bidra till projektmålet görs en jämförandestudie mellan den prediktiva noggrannheten hos en centraliseradmaskininlärningsalgoritm, som i nuläget används, och endistribuerad modell baserad på federerat lärande. Data somanvänds för träning och testning av båda modellerna samladesin genom en trådlös sensor utvecklad inom iWater-projektet.Den centraliserade algoritmen användes som grund för denutvecklade distribuerade modellen. På grund av brist på sensorersimulerades den distribuerade modellen genom nedprovtagningoch uppdelning av data i sex datamängder som representerarenskilda sensorer. Resultaten för båda modellerna var liknandeoch den utvecklade algoritmen har en noggrannhet på 98.41 % / Kandidatexjobb i elektroteknik 2021, KTH, Stockholm

With the rapid increase of Internet of Things-devices(IoT), demand for new machine learning algorithms and modelshas risen. The focus of this project is implementing a federatedlearning (FL) algorithm to detect anomalies in measurementsmade by a water monitoring IoT-sensor. The FL algorithm trainsacross a collection of decentralized IoT-devices, each using thelocal data acquired from the specific sensor. The local machinelearning models are then uploaded to a mutual server andaggregated into a global model. The global model is sent back tothe sensors and is used as a template when training starts againlocally. In this project, we only have had access to one physicalsensor. This has forced us to virtually simulate sensors. Thesimulation was done by splitting the data gathered by the onlyexisting sensor. To deal with the long, sequential data gatheredby the sensor, a long short-term memory (LSTM) network wasused. This is a special type of artificial neural network (ANN)capable of learning long-term dependencies. After analyzing theobtained results it became clear that FL has the potential toproduce good results, provided that more physical sensors aredeployed. / I samband med den snabba ökningen avInternet of Things-enheter (IoT) har efterfrågan på nya algoritmeroch modeller för maskininlärning ökat. Detta projektfokuserar på att implementera en federated learning (FL) algoritmför att detektera avvikelser i mätdata från en sensorsom övervakar vattenkvaliteten. FL algoritmen tränar en samlingdecentraliserade IoT-enheter, var och en med hjälp av lokaldata från sensorn i fråga. De lokala maskininlärningsmodellernaladdas upp till en gemensam server och sammanställs till englobal modell. Den globala modellen skickas sedan tillbaka tillsensorerna och används som mall när den lokala träningen börjarigen. I det här projektet hade vi endast tillgång till en fysisksensor. Vi har därför varit tvungna att simulera sensorer. Dettagjordes genom att dela upp datamängden som samlats in frånden fysiska sensorn. För att hantera den långa sekventiella dataanvänds ett long short-term memory (LSTM) nätverk. Detta ären speciell typ av artificiellt neuronnät (ANN) som är kapabeltatt minnas mönster under en längre tid. Efter att ha analyseratresultaten blev det tydligt att FL har potentialen att produceragoda resultat, givet att fler fysiska sensorer implementeras. / Kandidatexjobb i elektroteknik 2021, KTH, Stockholm