Global ETD Search

Search theses and dissertations gathered from participating repositories worldwide. Every result links back to the library that holds it. No account is needed.

Results

Showing 1 to 20 of 21 for “"Model parallelism"”.

  1. Exploring model parallelism in distributed scheduling of neural network frameworks

    … that can no longer fit on a single machine. A model parallel approach, backed by partitioning of Neural Networks and placement of operators on devices in a distributed system, provides a better distributed solution to this problem. In this thesis, we motivate the case for device placement in …

    uiuc Repository record for Exploring model parallelism in distributed scheduling of neural network frameworks (opens in a new tab)

  2. Distributed SLIDE: Enabling Training Large Neural Networks on Low Bandwidth and Simple CPU-Clusters via Model Parallelism and Sparsity

    … enable those CPU cycles to train heavyweight AI models. Our goal is against mainstream frameworks, which focus on leveraging expensive specialized ultra-high bandwidth interconnect to address the communication bottleneck in distributed neural network training. This paper presents a distributed …

    rice Repository record for Distributed SLIDE: Enabling Training Large Neural Networks on Low Bandwidth and Simple CPU-Clusters via Model Parallelism and Sparsity (opens in a new tab)

  3. Distributed Machine Learning Scale Out with Algorithms and Systems

    … inputs and outputs cannot fit on a GPU for model parallelism, or when a model cannot fit on a single machine for data parallelism. A TensorFlow code that works reasonably well on a single machine with eight GPUs procured from a cloud provider often runs slower on two machines totaling …

    rice Repository record for Distributed Machine Learning Scale Out with Algorithms and Systems (opens in a new tab)

  4. A survey of distributed learning in cloud, mobile, and edge settings

    … neural networks (CNNs), and large language models (LLMs), machine learning (ML) models are becoming increasingly complex, demanding significant computational resources for both inference and training stages. To address this challenge, distributed learning has emerged as a crucial approach, …

    texas Repository record for A survey of distributed learning in cloud, mobile, and edge settings (opens in a new tab)

  5. Automated Machine Learning with Constraints and Imperfect Data

    … to discover high-performance machine learning models automatically. It could release the burden of data scientists from the multifarious manual tuning process. However, dataset does not always have correct labels and sufficient data size. Wrong labels disrupt the training procedure and could …

    tamu Repository record for Automated Machine Learning with Constraints and Imperfect Data (opens in a new tab)

  6. Scaling Local Learning for Supervised and Self-supervised Learning

    … Local learning is an approach to model-parallelism that removes the standard end-to-end learning setup and utilizes local objective functions to permit parallel learning amongst model components in a deep network. Recent works have demonstrated that variants of local learning can …

    concordia Download PDF for Scaling Local Learning for Supervised and Self-supervised Learning (opens in a new tab)

  7. Efficient Distributed Training With Subnetworks

    … communication demands on modern accelerators. As model sizes continue to grow faster than available device memory, efficiently distributing training across multiple hardware devices has become increasingly challenging. In practice, this challenge is shaped not only by theoretical scaling limits …

    concordia Download PDF for Efficient Distributed Training With Subnetworks (opens in a new tab)

  8. Distributed convolutional neural network training on resource-constrained mobile and edge clusters

    … and expose communication and memory-aware parallelism. We also introduce the concept of layer grouping to further fine-tune performance based on computation and communication trade-off, and we describe an algorithm to determine the optimal grouping profile. Results show that for a cluster …

    texas Repository record for Distributed convolutional neural network training on resource-constrained mobile and edge clusters (opens in a new tab)

  9. Nonparametric Bayesian Deep Learning for Scientific Data Analysis

    … emerged as the leading paradigm for predictive modeling in a variety of domains, especially those involving large volumes of high-dimensional spatio-temporal data such as images and text. With the rise of big data in scientific and engineering problems, there is now considerable interest in the …

    utk-theses Repository record for Nonparametric Bayesian Deep Learning for Scientific Data Analysis (opens in a new tab)

  10. Deploying Deep Neural Networks in Edge with Distribution

    … inferences in real-time while exploiting several model-parallelism methods. Nonetheless, since communication is costly and current DNN models capture a single-chain of dependency pattern, distributing and parallelizing the computations of current DNNs may not be an effective solution for edge …

    gatech Repository record for Deploying Deep Neural Networks in Edge with Distribution (opens in a new tab)

  11. Mapping Neural Network Inference Onto Heterogeneous Hardware Platforms

    … by automatically leveraging both data and model parallelism. Furthermore, our modularity-based "splitting" heuristic significantly enhances solution runtime by up to 395$\times$, without compromising solution quality compared to the exact MILP approach. Additionally, it outperforms …

    cornell Repository record for Mapping Neural Network Inference Onto Heterogeneous Hardware Platforms (opens in a new tab)

  12. Device Placement Optimization with Deep Reinforcement Learning

    … study the device placement problem in a pipeline parallelism setting, where model parallelism and data parallelism are combined to speed up the DNN training. Specifically, we carefully redesigned the action space of the DRL agent for pipelined device placement, especially for the placement of …

    toronto-retro Repository record for Device Placement Optimization with Deep Reinforcement Learning (opens in a new tab)

  13. A General Method for Efficient Distributed Training and Federated Learning in Synchronous and Asynchronous Scenarios

    … learning systems, there is an eternal conflict: model performance versus model scale versus computation resources. The never ended desire to improve model performance significantly increases the size of machine learning model, the size of training dataset and the training time, while the …

    rice Repository record for A General Method for Efficient Distributed Training and Federated Learning in Synchronous and Asynchronous Scenarios (opens in a new tab)

  14. Distributed On-line Training for Object Detection on Embedded Devices.

    … distributed approach for object detection model training and inference, using low cost embedded devices. Examples of the usage of such an approach is automatic beach litter collection using mobile robots, IoT based intelligent video surveillance system for traffic monitoring, indoor …

    bournemouth Download PDF for Distributed On-line Training for Object Detection on Embedded Devices. (opens in a new tab)

  15. Efficiently Training Deep Learning Models on Elastic and Heterogeneous Cloud Resources

    … of serverless computing. By combining data and model parallelism with fine-grained resource provisioning, Hydrozoa achieves cost-effective training while eliminating cluster management overhead. Second, Cephalo addresses heterogeneous GPU clusters by independently balancing compute and memory …

    waterloo Repository record for Efficiently Training Deep Learning Models on Elastic and Heterogeneous Cloud Resources (opens in a new tab)

  16. OPTIMIZING PARALLELISM AND COMMUNICATION FOR LARGE-SCALE GENERATIVE DEEP LEARNING SYSTEMS

    … has shifted from discriminative to generative models capable of producing text, images, videos, and scientific insights. However, scaling models to hundreds of billions of parameters introduces compute, memory, communication, power, and development walls. This paper presents four systems …

    nus Repository record for OPTIMIZING PARALLELISM AND COMMUNICATION FOR LARGE-SCALE GENERATIVE DEEP LEARNING SYSTEMS (opens in a new tab) Download file for OPTIMIZING PARALLELISM AND COMMUNICATION FOR LARGE-SCALE GENERATIVE DEEP LEARNING SYSTEMS (opens in a new tab)

  17. Simulation and analysis of gradient coding protocols- Leveraging partial stragglers within gradient coding

    … essential for training modern deep learning models on large-scale datasets. However, heterogeneous worker performance in cloud computing environments that are characterized by variable computation speeds, network congestion, and hardware variability create significant bottlenecks in …

    iastate Repository record for Simulation and analysis of gradient coding protocols- Leveraging partial stragglers within gradient coding (opens in a new tab)

  18. Real-time guarantees in distributed edge computing

    … be utilized to improve performance by exploiting parallelism. Traditionally, this has been applied to target improved throughput. However, in real-time applications, latency is a key concern. Furthermore, in many cases, such as the Internet of Things (IoT), such distribution occurs over open …

    texas Repository record for Real-time guarantees in distributed edge computing (opens in a new tab)

  19. Randomized Algorithms for Training Deep Models with Large Outputs

    … AI tasks, especially in NLP, can be naturally modeled as extreme classification problems leading to improved precision. However, such models are prohibitively expensive to train due to the memory blow-up in the last layer. As an example, we will delve into a real Amazon Search dataset, for …

    rice Repository record for Randomized Algorithms for Training Deep Models with Large Outputs (opens in a new tab)

  20. Adaptive optimization of intelligent agents

    … innovation paves the way for novel real-life model-parallelism implementations that solely require asynchronous updates between partitions since the communicational overhead prohibits the efficient training. A few examples may concern complex, geographically dispersed IoT ecosystems, …

    greece Repository record for Adaptive optimization of intelligent agents (opens in a new tab)

Page 1 of 2