<p dir="ltr">Sparse basis recovery is an important learning problem when the number of model dimensions (<i>p</i>) is much larger than the number of samples (<i>n</i>). However, there has been little work that studies sparse basis recovery in the Federated Learning (FL) setting, where the Differential Privacy (DP) of the client data must also be simultaneously protected. Notably, the performance guarantees of existing DP-FL algorithms (such as DP-SGD) will degrade significantly when the system is ill-determined (i.e., <i>p >> n</i>), and thus they will fail to accurately learn the true underlying sparse model. The goal of my thesis is therefore to develop DP-FL sparse basis recovery algorithms that can recover the true underlying sparse basis provably accurately even when <i>p >> n</i>, yet still guaranteeing the differential privacy of the client data.</p><p dir="ltr">During my PhD studies, we developed three DP-FL sparse basis recovery algorithms for this purpose. Our first algorithm, SPriFed-OMP, based on the Orthogonal Matching Pursuit (OMP) algorithm, can achieve high accuracy even when <i>n = O(\sqrt{p})</i> under the stronger Restricted Isometry Property (RIP) assumption for least-square problems. Our second algorithm, Humming-Bird, based on a carefully modified variant of the Forward-Backward Algorithm (FoBA), can achieve differentially private sparse recovery for the same setup while requiring the much weaker Restricted Strong Convexity (RSC) condition. We further extend Humming-Bird to support loss functions beyond least-square satisfying the RSC condition. To the best of our knowledge, these are the first DP-FL results guaranteeing sparse basis recovery in the <i>p >> n</i> setting.</p>

The distributed and privacy sensitive nature of cellular networks make them strong candidates for optimization using Federated Learning, but this exposes them to a problem inherent to the learning paradigm: performance inequality due to heterogeneous client data distributions. The prevailing approach of enforcing uniform client performance ignores client-specific performance limitations due to different levels of irreducible uncertainty present in their data, resulting in deteriorated network performance. To address this issue, this thesis introduces two novel federated algorithms designed to enhance learning efficiency and ensure fairness in the presence of heteroscedastic noise, reflecting the distributive justice principles of utilitarianism and equality. Under these circumstances, the proposed algorithms are shown to significantly improve overall performance and performance fairness. The deployment of these algorithms promises a dual benefit: enhancement in network performance and a fairer distribution of service quality for end users.

The future quantum Internet (QI) will transform today's communication networks and user experiences by providing unparalleled security levels, superior quantum computational powers, along with enhanced sensing accuracy and data processing capabilities. These features will be enabled through applications like quantum key distribution (QKD) and quantum machine learning (QML). Towards enabling these applications, the QI requires the development of global quantum communication networks (QCNs) that enable the distribution of entangled resources between distant nodes. This dissertation addresses two major challenges facing QCNs, which are the scalability and coverage of their architectures, and the efficiency of their operations. Additionally, the dissertation studies the near-term deployment of QML applications over today's noisy quantum devices, essential for realizing the future QI. In doing so, the scalability and efficiency challenges facing the different QCN elements are explored, and practical noise-aware and physics-informed approaches are developed to optimize the QCN performance given heterogeneous quantum application-specific quality of service (QoS) user requirements on entanglement rate and fidelity. Towards achieving this goal, this dissertation makes a number of key contributions. First, the scaling limits of quantum repeaters is investigated, and a holistic optimization framework is proposed to optimize the geographical coverage of quantum repeater networks (QRNs), including the number of quantum repeaters, their placement and separating distances, quantum memory management, and quantum operations scheduling. Then, a novel framework is proposed to address the scalability challenge of free-space optical (FSO) quantum channels in the presence of blockages and environmental effects. Particularly, the utilization of a reconfigurable intelligent surface (RIS) in QCNs is proposed to maintain a line-of-sight (LoS) connection between quantum nodes separated by blockages, and a novel analytical model of quantum noise and end-to-end (e2e) fidelity in such QCNs is developed. The results show enhanced entangled state fidelity and entanglement distribution rates, improving user fairness by around 40% compared to benchmark approaches. The dissertation then investigates the efficiency challenges in a practical use-case of QCNs with a single quantum switch (QS). Particularly, the average quantum memory noise effects are analytically analyzed and their impacts on the allocation of entanglement generation sources and minimization of entanglement distribution delay while optimizing QS entanglement distillation operations are investigated. The results show an enhanced e2e fidelity and a minimized e2e entanglement distribution delay compared to existing approaches, and a unique capability of satisfying all users QoS requirements. This QCN architecture is scaled up with multiple QSs serving heterogeneous user requests, necessary for scalable quantum applications over the QI. Here, a novel efficient matching theory-based framework for optimizing the request-QS association in such QCNs while managing quantum memories and optimizing QS operations is proposed. Finally, after scaling QCNs and ensuring their efficient operations, the dissertation proposes novel distributed QML frameworks that can leverage both classical networks and QCNs to enable collaborative learning between today's noisy quantum devices. In particular, the first quantum federated learning (QFL) frameworks incorporating different quantum neural networks and leveraging quantum and classical data are developed, and the first publicly available federated quantum dataset is introduced. The results show enhanced performance and reductions in the communication overhead and number of training epochs needed until convergence, compared to classical counterpart frameworks. Overall, this dissertation develops robust frameworks and algorithms that advance the theoretical understanding of QCNs and offers practical insights for the future development of the QI and its applications. The dissertation concludes by analyzing some open challenges facing QCNs and proposing a vision for physics-informed QCNs, along with important future directions. / Doctor of Philosophy / In today's digital age, we are generating vast amounts of data through videos, live streams, and various online activities. This explosion of data brings not only incredible opportunities for innovation but also heightened security concerns. The current Internet infrastructure struggles to keep up with the demand for speed and security. In this regard, the quantum Internet (QI) emerges as a revolutionary technology poised to make the communication and data sharing processes faster and more secure than ever before. The QI requires the development of quantum communication networks (QCNs) that will be seamlessly integrated with today's existing communication systems that form today's Internet. This way, the QI enables ultra-secure communication and advanced computing applications that can transform various sectors, from finance to healthcare. However, building such global QCNs, requires overcoming significant challenges, including the sensitive nature and limitations of quantum devices. In this regard, the goal of this dissertation is to develop scalable and efficient QCNs that overcome the different challenges facing different QCN elements and enable a wide coverage and robust performance towards realizing the QI at a global scale. Simultaneously, machine learning (ML), which is driving significant advancements and transforming industries in today's world. Here, quantum technologies are anticipated to make a breakthrough in ML through quantum machine learning (QML) models that can handle today's large and complex data. However, quantum computers are still limited in scale and efficiency, often being noisy and unreliable. Throughout this dissertation, these limitations of QML are addressed by developing frameworks that allow multiple quantum computers to work together collaboratively in a distributed manner over classical networks and QCNs. By leveraging distributed QML, it is possible to achieve remarkable advancements in privacy and data utilization. For instance, distributed QML can enhance navigation systems by providing more accurate and secure route planning or revolutionize healthcare by enabling secure and efficient analysis of medical data. In summary, this dissertation addresses the critical challenges of building scalable and efficient QCNs to support the QI and develops distributed QML frameworks to enable near-term utilization of QML in transformative applications. By doing so, it paves the way for a future where quantum technology is integral to our daily lives, enhancing security, efficiency, and innovation across various domains.

Decentralized Machine Learning could address some problematic facets with Federated Learning. There is no central server acting as an arbiter of whom or what may benefit from Machine Learning models created by the vast amount of data becoming available in recent years. It could also increase the reliability and scalability of Machine Learning systems thereby drawing the benefit of having more data accessible. Gossip Learning is such a protocol, but has primarily been designed with linear models in mind. How does Gossip Learning perform when training Deep Neural Networks? Could it be a viable alternative to Federated Learning? In this thesis, we implement Gossip Learning using two different model merging strategies. We also design and implement two extensions to this protocol with the goal of achieving higher performance when training under churn. The training methods are compared on two tasks: image classification on the Federated Extended MNIST dataset and time- series forecasting on the NN5 dataset. Additionally, we also run an experiment where learners churn, alternating between being available and unavailable. We find that Gossip Learning performs slightly better in settings where learners do not churn but is vastly outperformed in the setting where they do. / Decentraliserad Maskinginlärning kan lösa några problematiska aspekter med Federated Learning. Det finns ingen central server som agerar som domare för vilka som får gagna av Maskininlärningsmodellerna skapad av den stora mäng data som blivit tillgänglig på senare år. Det skulle också kunna öka pålitligheten och skalbarheten av Maskininlärningssystem och därav dra nytta av att mer data är tillgänglig. Gossip Learning är ett sånt protokoll, men det är primärt designat med linjära modeller i åtanke. Hur presterar Gossip Learning när man tränar Djupa Neurala Nätverk? Kan det vara ett möjligt alternativ till Federated Learning? I det här exjobbet implementerar vi Gossip Learning med två olika modelsammanslagningstekniker. Vi designar och implementerar även två tillägg till protokollet med målet att uppnå bättre prestanda när man tränar i system där noder går ner och kommer up. Träningsmetoderna jämförs på två uppgifter: bildklassificering på Federated Extended MNIST datauppsättningen och tidsserieprognostisering på NN5 datauppsättningen. Dessutom har vi även experiment då noder alternerar mellan att vara tillgängliga och otillgängliga. Vi finner att Gossip Learning presterar marginellt bättre i miljöer då noder alltid är tillgängliga men är kraftigt överträffade i miljöer då noder alternerar mellan att vara tillgängliga och otillgängliga.

Federated learning faces a challenge when dealing with highly heterogeneous data and it can sometimes be inadequate to adopt an approach where a single model is trained for usage at all nodes in the network. Different approaches have been investigated to succumb this issue such as adapting the trained model to each node and clustering the nodes in the network and train a different model for each cluster where the data is less heterogeneous. In this work we study the possibilities to improve the local model performance utilizing the hierarchical setup that comes with clustering the participating clients in the network. Experiments are carried out featuring a Long Short-Term Memory network to perform time series forecasting to evaluate different approaches utilizing the hierarchical setup and comparing them to standard federated learning approaches. The experiments are done using a dataset collected by Ericsson AB consisting of handovers recorded at base stations in an European city. The hierarchical approaches didn’t show any benefit over common two-level approaches. / Federated Learning står inför en utmaning när det gäller att hantera data med en hög grad av heterogenitet och det kan i vissa fall vara olämpligt att använda sig av en approach där en och samma modell är tränad för att användas av alla noder i nätverket. Olika approacher för att hantera detta problem har undersökts som att anpassa den tränade modellen till varje nod och att klustra noderna i nätverket och träna en egen modell för varje kluster inom vilket datan är mindre heterogen. I detta arbete studeras möjligheterna att förbättra prestandan hos de lokala modellerna genom att dra nytta av den hierarkiska anordning som uppstår när de deltagande noderna i nätverket grupperas i kluster. Experiment är utförda med ett Long Short-Term Memory-nätverk för att utföra tidsserieprognoser för att utvärdera olika approacher som drar nytta av den hierarkiska anordningen och jämför dem med vanliga federated learning-approacher. Experimenten är utförda med ett dataset insamlat av Ericsson AB. Det består av "handoversfrån basstationer i en europeisk stad. De hierarkiska approacherna visade inga fördelar jämfört med de vanliga två-nivåapproacherna.

No description available.

<p>Electric load forecasting is a critical tool for power system planning and the creation of sustainable energy systems. Precise and reliable load forecasting enables power system operators to make informed decisions regarding power generation and transmission, optimize energy efficiency, and reduce operational costs and extra power generation costs, to further reduce environment-related issues. However, achieving desirable forecasting performance remains challenging due to the irregular, nonstationary, nonlinear, and noisy nature of the observed data under unprecedented events. In recent years, deep learning and other artificial intelligence techniques have emerged as promising approaches for load forecasting. These techniques have the ability to capture complex patterns and relationships in the data and adapt to changing conditions, thereby enhancing forecasting accuracy. As such, the use of deep learning and other artificial intelligence techniques in load forecasting has become an increasingly popular research topic in the field of power systems. </p> <p>Although deep learning techniques have advanced load forecasting, the field still requires more accurate and efficient models. One promising approach is federated learning, which allows for distributed data analysis without exchanging data among multiple devices or cen- ters. This method is particularly relevant for load forecasting, where each power station’s data is sensitive and must be protected. In this study, a proposed approach utilizing Federated Deeponet for seven different power stations is introduced, which proposes a Federated Deep Operator Networks and a Lagevin Dynamics-based Federated Deep Operator Networks using Stochastic Gradient Langevin Dynamics as the optimizer for training the data daily for one-day and predicting for one-day frequencies by frequencies. The data evaluation methods include mean absolute percentage error and the percentage coverage under confidence interval. The findings demonstrate the potential of federated learning for secure and precise load forecasting, while also highlighting the challenges and opportunities of implementing this approach in real-world scenarios. </p>

<p>This dissertation explore two major goals in Deep Learning algorithm design: efficiency and trustworthiness. We motivate these concerns in Chapter 1 and give relevant background in Chapter 2. We then discuss six works to target these two goals. </p> <p>The first of these discusses how to make the model compression methodology more efficient, so it can be done in a single shot. This allows us to create models with reduced size and layers, so we can have faster and more efficient inference, and is covered in Chapter 3. We then extend this to target efficiency in continual learning in Chapter 4, while mitigating the problem of catastrophic forgetting. The method discussed also allows us to circumvent the potential for data leakage by avoiding the need to store any data from the past tasks. Next, we consider brain-inspired computing as an alternative to traditional neural networks to improve compute efficiency of networks. The spiking neural networks discussed however have large inference latency due to the need for accumulating spikes over many timesteps. We tackle this by introducing a new scheme that distributes an image over time by breaking it down into a sum of its ranked sinusoidal bases in Chapter 5. This results in networks that are faster and more efficient to deploy. Chapter 6 targets mitigating both the communication expense and potential for data leakage in federated learning, by distilling the gradients to be communicated in a small number of images that resemble noise. Communicating these images is more efficient, and circumvents the potential for data leakage as they resemble noise. We then explore the applications of studying curvature of loss with respect to input data points in the last two chapters. We first utilize curvature to create performant coresets to reduce the size of datasets, to make training more efficient in Chapter 7. In Chapter 8, we use curvature as a metric for overfitting and use it to expose dataset integrity issues arising from memorization.</p>

Natural Language Processing (NLP) is one of the most popular and visible forms of Artificial Intelligence in recent years. This is partly because it has to do with a common characteristic of human beings: language. NLP applications allow to create new services in the industrial sector in order to offer new solutions and provide significant productivity gains. All of this has happened thanks to the rapid progression of Deep Learning models. Large scale contextual representation models, such asWord2Vec, ELMo and BERT, have significantly advanced NLP in recently years. With these latest NLP models, it is possible to understand the semantics of text to a degree never seen before. However, they require large amounts of text data to process to achieve high-quality results. This data can be gathered from different sources, but one of the main collection points are devices such as smartphones, smart appliances and smart sensors. Lamentably, joining and accessing all this data from multiple sources is extremely challenging due to privacy and regulatory reasons. New protocols and techniques have been developed to solve this limitation by training models in a massively distributed manner taking advantage of the powerful characteristic of the devices that generates the data. Particularly, this research aims to test the viability of training NLP models, in specific Word2Vec, with a massively distributed protocol like Federated Learning. The results show that FederatedWord2Vecworks as good as Word2Vec is most of the scenarios, even surpassing it in some semantics benchmark tasks. It is a novel area of research, where few studies have been conducted, with a large knowledge gap to fill in future researches. / Naturlig språkbehandling är en av de mest populära och synliga formerna av artificiell intelligens under de senaste åren. Det beror delvis på att det har att göra med en gemensam egenskap hos människor: språk. Naturlig språkbehandling applikationer gör det möjligt att skapa nya tjänster inom industrisektorn för att erbjuda nya lösningar och ge betydande produktivitetsvinster. Allt detta har hänt tack vare den snabba utvecklingen av modeller för djup inlärning. Modeller i storskaligt sammanhang, som Word2Vec, ELMo och BERT har väsentligt avancerat naturligt språkbehandling på senare tid år. Med dessa senaste naturliga språkbearbetningsmo modeller är det möjligt att förstå textens semantik i en grad som aldrig sett förut. De kräver dock stora mängder textdata för att bearbeta för att uppnå högkvalitativa resultat. Denna information kan samlas in från olika källor, men ett av de viktigaste insamlingsställena är enheter som smartphones, smarta apparater och smarta sensorer. Beklagligtvis är det extremt utmanande att gå med och komma åt alla dessa uppgifter från flera källor på grund av integritetsskäl och regleringsskäl. Nya protokoll och tekniker har utvecklats för att lösa denna begränsning genom att träna modeller på ett massivt distribuerat sätt med fördel av de kraftfulla egenskaperna hos enheterna som genererar data. Särskilt syftar denna forskning till att testa livskraften för att utbilda naturligt språkbehandling modeller, i specifika Word2Vec, med ett massivt distribuerat protokoll som Förenat Lärande. Resultaten visar att det Förenade Word2Vec fungerar lika bra som Word2Vec är de flesta av scenarierna, till och med överträffar det i vissa semantiska riktmärken. Det är ett nytt forskningsområde, där få studier har genomförts, med ett stort kunskapsgap för att fylla i framtida forskningar.

This study explored how clients with distorted data affected the Federated Learning process using the FedAvg and FedProx algorithms. Different amounts of the three distortions, Translation, Rotation, and Blur, were tested using three different Machine Learning models. The models were a Dense network, the well-known convolutional network LeNet-5, and a smaller version of the ResNet architecture. The results of the study successfully showcases how different distortions affect the three models. Therefore, they also show that the risk of local data distortion is important to factor in when picking a Machine Learning model for Federated Learning.