Back to search

University of Washington

Automated Parallelization to Improve Usability and Efficiency of Distributed Neural Network Training

Abstract

dc:description.abstract

The recent success of Deep Neural Networks (DNNs) [1] has triggered a race to build larger and larger DNNs [2]; however, a known limitation is the training speed [3]. To solve this speed problem, distributed neural network training has become an increasingly large area of research [4], [5]. Usability, the complexity for a machine learning or data scientist to implement distributed neural network training, is an aspect rarely considered, yet critical. There is strong evidence growing complexity has a direct impact on development effort, maintainability, and fault proneness of software [6]–[8]. We investigated, if automation can greatly reduce the implementation complexity of distributing neural network training across multiple devices without loss of computational efficiency when compared to manual parallelization. Experiments were conducted using Convolutional Neural Networks (CNN) and Multi-Layer Perceptron (MLP) networks to perform image classification on CIFAR-10 and MNIST datasets. Hardware consisted of an embedded, four node NVIDIA Jetson TX1 cluster. Torch Automatic Distributed Neural Network (TorchAD-NN) reduces the implementation complexity of data parallel neural network training by more than 90% and providing components, with near zero implementation complexity, to easily parallelize all or only select fully-connected neural layers.

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Grabaskas, Nathaniel J
Advisor dc:contributor.advisor
  • Fukuda, Munehiro

Subjects

dc:subject × 6

Rights

dc:rights
Statement dc:rights
  • CC BY
Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
http://hdl.handle.net/1773/41725

Chain of custody

source
Harvested from
University of Washington
Base URL
digital.lib.washington.edu/server/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Grabaskas, Nathaniel J. Automated Parallelization to Improve Usability and Efficiency of Distributed Neural Network Training. 2018. http://hdl.handle.net/1773/41725