<p dir="ltr">Sparse basis recovery is an important learning problem when the number of model dimensions (<i>p</i>) is much larger than the number of samples (<i>n</i>). However, there has been little work that studies sparse basis recovery in the Federated Learning (FL) setting, where the Differential Privacy (DP) of the client data must also be simultaneously protected. Notably, the performance guarantees of existing DP-FL algorithms (such as DP-SGD) will degrade significantly when the system is ill-determined (i.e., <i>p >> n</i>), and thus they will fail to accurately learn the true underlying sparse model. The goal of my thesis is therefore to develop DP-FL sparse basis recovery algorithms that can recover the true underlying sparse basis provably accurately even when <i>p >> n</i>, yet still guaranteeing the differential privacy of the client data.</p><p dir="ltr">During my PhD studies, we developed three DP-FL sparse basis recovery algorithms for this purpose. Our first algorithm, SPriFed-OMP, based on the Orthogonal Matching Pursuit (OMP) algorithm, can achieve high accuracy even when <i>n = O(\sqrt{p})</i> under the stronger Restricted Isometry Property (RIP) assumption for least-square problems. Our second algorithm, Humming-Bird, based on a carefully modified variant of the Forward-Backward Algorithm (FoBA), can achieve differentially private sparse recovery for the same setup while requiring the much weaker Restricted Strong Convexity (RSC) condition. We further extend Humming-Bird to support loss functions beyond least-square satisfying the RSC condition. To the best of our knowledge, these are the first DP-FL results guaranteeing sparse basis recovery in the <i>p >> n</i> setting.</p>

The distributed and privacy sensitive nature of cellular networks make them strong candidates for optimization using Federated Learning, but this exposes them to a problem inherent to the learning paradigm: performance inequality due to heterogeneous client data distributions. The prevailing approach of enforcing uniform client performance ignores client-specific performance limitations due to different levels of irreducible uncertainty present in their data, resulting in deteriorated network performance. To address this issue, this thesis introduces two novel federated algorithms designed to enhance learning efficiency and ensure fairness in the presence of heteroscedastic noise, reflecting the distributive justice principles of utilitarianism and equality. Under these circumstances, the proposed algorithms are shown to significantly improve overall performance and performance fairness. The deployment of these algorithms promises a dual benefit: enhancement in network performance and a fairer distribution of service quality for end users.

京都大学 / 新制・課程博士 / 博士(情報学) / 甲第25427号 / 情博第865号 / 新制||情||145(附属図書館) / 京都大学大学院情報学研究科社会情報学専攻 / (主査)教授 伊藤 孝行, 教授 黒田 知宏, 教授 岡部 寿男, 吉川 正俊(京都大学 名誉教授) / 学位規則第4条第1項該当 / Doctor of Informatics / Kyoto University / DFAM

Decentralized Machine Learning could address some problematic facets with Federated Learning. There is no central server acting as an arbiter of whom or what may benefit from Machine Learning models created by the vast amount of data becoming available in recent years. It could also increase the reliability and scalability of Machine Learning systems thereby drawing the benefit of having more data accessible. Gossip Learning is such a protocol, but has primarily been designed with linear models in mind. How does Gossip Learning perform when training Deep Neural Networks? Could it be a viable alternative to Federated Learning? In this thesis, we implement Gossip Learning using two different model merging strategies. We also design and implement two extensions to this protocol with the goal of achieving higher performance when training under churn. The training methods are compared on two tasks: image classification on the Federated Extended MNIST dataset and time- series forecasting on the NN5 dataset. Additionally, we also run an experiment where learners churn, alternating between being available and unavailable. We find that Gossip Learning performs slightly better in settings where learners do not churn but is vastly outperformed in the setting where they do. / Decentraliserad Maskinginlärning kan lösa några problematiska aspekter med Federated Learning. Det finns ingen central server som agerar som domare för vilka som får gagna av Maskininlärningsmodellerna skapad av den stora mäng data som blivit tillgänglig på senare år. Det skulle också kunna öka pålitligheten och skalbarheten av Maskininlärningssystem och därav dra nytta av att mer data är tillgänglig. Gossip Learning är ett sånt protokoll, men det är primärt designat med linjära modeller i åtanke. Hur presterar Gossip Learning när man tränar Djupa Neurala Nätverk? Kan det vara ett möjligt alternativ till Federated Learning? I det här exjobbet implementerar vi Gossip Learning med två olika modelsammanslagningstekniker. Vi designar och implementerar även två tillägg till protokollet med målet att uppnå bättre prestanda när man tränar i system där noder går ner och kommer up. Träningsmetoderna jämförs på två uppgifter: bildklassificering på Federated Extended MNIST datauppsättningen och tidsserieprognostisering på NN5 datauppsättningen. Dessutom har vi även experiment då noder alternerar mellan att vara tillgängliga och otillgängliga. Vi finner att Gossip Learning presterar marginellt bättre i miljöer då noder alltid är tillgängliga men är kraftigt överträffade i miljöer då noder alternerar mellan att vara tillgängliga och otillgängliga.

No description available.

The increasing complexity of trading strategies, when combined with machine learning models, forces market surveillance corporations to develop increasingly sophisticated methods for recognizing potential misuse. One strategy is to employ traders’ weapons against themselves, namely machine learning. However, the data utilized in market surveillance is highly sensitive, what may be available for machine learning is limited. In this thesis, we examine how federated learning for time series data can be used to identify potential market abuse while maintaining client privacy and data security. We are interested in developing a time-series-specific neural network employing federated learning. We demonstrate that when this strategy is used, the performance of detecting potential market abuse is comparable to that of the standard data centralized approach. Specifically, a non-federated model, a federated model, and a federated model with extra data privacy and security protection are evaluated and compared. Each model utilize an LSTM autoencoder to identify market abuse. The results demonstrate that a federated model’s performance in detecting possible market abuse is comparable to that of a non-federated model. Moreover, a federated approach with extra data privacy and security experienced a slight performance loss but is still a competitive model in comparison to the other models. Although this approach results in increased privacy and security, there is a limit to how much privacy and security can be ensured, as excessive privacy led to extremely poor performance. Federated learning offers the ability to increase data privacy and security with little performance decrease. / Den ökande komplexiteten handelsstrategier, i kombination med maskininlärning modeller, tvingar marknadsövervakning företag att utveckla allt mer sofistikerade metoder för att identifiera potentiellt marknadsmissbruk. En strategi är att använda handlarnas vapen mot sig själva, nämligen maskininlärning. Däremot, data som används inom marknadsövervakning är mycket känslig och vad som kan finnas tillgängligt för maskininlärning är begränsat.I den här studien undersöker vi hur federerat lärande för tidsseriedata kan användas till att identifiera potentiellt marknadsmissbruk samtidigt som klienternas integritet och datasäkerhet bibehålls. Vi är intresserade av att utveckla ett tidsserie-specifikt neuralt nätverk med hjälp av federated inlärning. Vi visar att när denna strategi används är prestanda för att upptäcka potentiellt marknadsmissbruk jämförbart med det för den vanliga data-centraliserade metoden. Specifikt, en icke-federerad modell, en federerad modell och en federerad modell med extra dataintegritet och säkerhet utvärderas och jämförs. Varje modell använder en LSTM-Autoencoder för att identifiera marknadsmissbruk. Resultaten visar att en federerad modells prestanda när det gäller att upptäcka eventuellt marknadsmissbruk är jämförbar med en icke-federerad modell. Dessutom, ett federerat tillvägagångssätt med extra dataintegritet upplevde en liten prestandaförlust men är fortfarande en konkurrenskraftig modell i jämförelse med andra modeller. Även om detta tillvägagångssätt resulterar i ökad integritet och säkerhet, finns det en gräns för hur mycket som kan säkerställas. Federated learning möjliggör ökad datasekretess och säkerhet med liten prestandasänkning.

<p>Electric load forecasting is a critical tool for power system planning and the creation of sustainable energy systems. Precise and reliable load forecasting enables power system operators to make informed decisions regarding power generation and transmission, optimize energy efficiency, and reduce operational costs and extra power generation costs, to further reduce environment-related issues. However, achieving desirable forecasting performance remains challenging due to the irregular, nonstationary, nonlinear, and noisy nature of the observed data under unprecedented events. In recent years, deep learning and other artificial intelligence techniques have emerged as promising approaches for load forecasting. These techniques have the ability to capture complex patterns and relationships in the data and adapt to changing conditions, thereby enhancing forecasting accuracy. As such, the use of deep learning and other artificial intelligence techniques in load forecasting has become an increasingly popular research topic in the field of power systems. </p> <p>Although deep learning techniques have advanced load forecasting, the field still requires more accurate and efficient models. One promising approach is federated learning, which allows for distributed data analysis without exchanging data among multiple devices or cen- ters. This method is particularly relevant for load forecasting, where each power station’s data is sensitive and must be protected. In this study, a proposed approach utilizing Federated Deeponet for seven different power stations is introduced, which proposes a Federated Deep Operator Networks and a Lagevin Dynamics-based Federated Deep Operator Networks using Stochastic Gradient Langevin Dynamics as the optimizer for training the data daily for one-day and predicting for one-day frequencies by frequencies. The data evaluation methods include mean absolute percentage error and the percentage coverage under confidence interval. The findings demonstrate the potential of federated learning for secure and precise load forecasting, while also highlighting the challenges and opportunities of implementing this approach in real-world scenarios. </p>

<p>This dissertation explore two major goals in Deep Learning algorithm design: efficiency and trustworthiness. We motivate these concerns in Chapter 1 and give relevant background in Chapter 2. We then discuss six works to target these two goals. </p> <p>The first of these discusses how to make the model compression methodology more efficient, so it can be done in a single shot. This allows us to create models with reduced size and layers, so we can have faster and more efficient inference, and is covered in Chapter 3. We then extend this to target efficiency in continual learning in Chapter 4, while mitigating the problem of catastrophic forgetting. The method discussed also allows us to circumvent the potential for data leakage by avoiding the need to store any data from the past tasks. Next, we consider brain-inspired computing as an alternative to traditional neural networks to improve compute efficiency of networks. The spiking neural networks discussed however have large inference latency due to the need for accumulating spikes over many timesteps. We tackle this by introducing a new scheme that distributes an image over time by breaking it down into a sum of its ranked sinusoidal bases in Chapter 5. This results in networks that are faster and more efficient to deploy. Chapter 6 targets mitigating both the communication expense and potential for data leakage in federated learning, by distilling the gradients to be communicated in a small number of images that resemble noise. Communicating these images is more efficient, and circumvents the potential for data leakage as they resemble noise. We then explore the applications of studying curvature of loss with respect to input data points in the last two chapters. We first utilize curvature to create performant coresets to reduce the size of datasets, to make training more efficient in Chapter 7. In Chapter 8, we use curvature as a metric for overfitting and use it to expose dataset integrity issues arising from memorization.</p>

Natural Language Processing (NLP) is one of the most popular and visible forms of Artificial Intelligence in recent years. This is partly because it has to do with a common characteristic of human beings: language. NLP applications allow to create new services in the industrial sector in order to offer new solutions and provide significant productivity gains. All of this has happened thanks to the rapid progression of Deep Learning models. Large scale contextual representation models, such asWord2Vec, ELMo and BERT, have significantly advanced NLP in recently years. With these latest NLP models, it is possible to understand the semantics of text to a degree never seen before. However, they require large amounts of text data to process to achieve high-quality results. This data can be gathered from different sources, but one of the main collection points are devices such as smartphones, smart appliances and smart sensors. Lamentably, joining and accessing all this data from multiple sources is extremely challenging due to privacy and regulatory reasons. New protocols and techniques have been developed to solve this limitation by training models in a massively distributed manner taking advantage of the powerful characteristic of the devices that generates the data. Particularly, this research aims to test the viability of training NLP models, in specific Word2Vec, with a massively distributed protocol like Federated Learning. The results show that FederatedWord2Vecworks as good as Word2Vec is most of the scenarios, even surpassing it in some semantics benchmark tasks. It is a novel area of research, where few studies have been conducted, with a large knowledge gap to fill in future researches. / Naturlig språkbehandling är en av de mest populära och synliga formerna av artificiell intelligens under de senaste åren. Det beror delvis på att det har att göra med en gemensam egenskap hos människor: språk. Naturlig språkbehandling applikationer gör det möjligt att skapa nya tjänster inom industrisektorn för att erbjuda nya lösningar och ge betydande produktivitetsvinster. Allt detta har hänt tack vare den snabba utvecklingen av modeller för djup inlärning. Modeller i storskaligt sammanhang, som Word2Vec, ELMo och BERT har väsentligt avancerat naturligt språkbehandling på senare tid år. Med dessa senaste naturliga språkbearbetningsmo modeller är det möjligt att förstå textens semantik i en grad som aldrig sett förut. De kräver dock stora mängder textdata för att bearbeta för att uppnå högkvalitativa resultat. Denna information kan samlas in från olika källor, men ett av de viktigaste insamlingsställena är enheter som smartphones, smarta apparater och smarta sensorer. Beklagligtvis är det extremt utmanande att gå med och komma åt alla dessa uppgifter från flera källor på grund av integritetsskäl och regleringsskäl. Nya protokoll och tekniker har utvecklats för att lösa denna begränsning genom att träna modeller på ett massivt distribuerat sätt med fördel av de kraftfulla egenskaperna hos enheterna som genererar data. Särskilt syftar denna forskning till att testa livskraften för att utbilda naturligt språkbehandling modeller, i specifika Word2Vec, med ett massivt distribuerat protokoll som Förenat Lärande. Resultaten visar att det Förenade Word2Vec fungerar lika bra som Word2Vec är de flesta av scenarierna, till och med överträffar det i vissa semantiska riktmärken. Det är ett nytt forskningsområde, där få studier har genomförts, med ett stort kunskapsgap för att fylla i framtida forskningar.

Federated Learning now a days has emerged as a promising standard for machine learning model training, which can be executed collaboratively on decentralized data sources. As the adoption of Federated Learning grows, the selection of the apt frame work for our use case has become more important. This study mainly concentrates on a comprehensive overview of three prominent Federated Learning frameworks Flower, FedN, and FedML. The performance, scalability, and utilization these frame works is assessed on the basis of an NLP use case. The study commences with an overview of Federated Learning and its significance in distributed learning scenarios. Later on, we explore into the examination of the Flower framework in-depth covering its structure, communication methods and interaction with deep learning libraries. The performance of Flower is evaluated by conducting experiments on a standard benchmark dataset. Metrics provide measurements for accuracy, speed and scalability. Tests are also conducted to assess Flower's ability to handle large-scale Federated Learning setups. The same is carried out with the other two frameworks FedN and FedML also. To gain better insight into the strengths, limitations, and suitability of Flower, FedN, and FedML for different Federated Learning scenarios, the study utilizes the above stated comparative analysis on a real time use case. The possibilities for integrating these frameworks with current machine learning workflows are discussed. Furthermore, the final results and conclusions may help researchers and practitioners to make conversant decisions regarding framework selection for their Federated Learning applications. / <p>Det finns övrigt digitalt material (t.ex. film-, bild- eller ljudfiler) eller modeller/artefakter tillhörande examensarbetet som ska skickas till arkivet.</p><p>There are other digital material (eg film, image or audio files) or models/artifacts that belongs to the thesis and need to be archived.</p>