Back to search

University of Leeds

Accelerating Deep Learning by Optimising Communication and Computation Efficiency

Abstract

dc:description.abstract

Deep Neural Networks (DNNs) underpin many of today’s business and scientific innovations. However, realising the full potential of DNNs and making them more accessible requires tackling multiple challenges across the computing stack. These include improving the efficiency of distributed communication across computing devices and computation on a single computing node. This thesis presents new optimisations for DNN training across multiple distributed machines and on a single machine. Our techniques reduce DNN training times, making DNNs faster and more efficient. First, we improve the communication efficiency of distributed training through multi-streamed gradient communication. Our approach allows a training worker to participate in multiple gradient communication operations simultaneously to improve network bandwidth utilisation and reduce communication latency, achieving up to 13.4x training throughput compared to existing distributed training frameworks. Next, we develop new techniques to improve DNNs’ performance on single machines, using Graph Neural Networks (GNNs) as a case study. While GNNs are a powerful tool for processing graph data, their scalability is hindered by the large memory footprint typically required to process real-life datasets. To mitigate this issue, we leverage two kernel-based memory-saving strategies: lossless sparse matrix storage and Tensor-train Decomposition (TTD), which trades accuracy for memory efficiency. Our goal is to accelerate the computation of these memory-saving techniques. For sparse matrix storage, we enhance GNN kernels by introducing a machine-learning-based method to dynamically select the optimal sparse storage format based on the input data, improving sparse GNN kernels by up to 3x. To accelerate TTD-based GNNs, we optimise data reuse and kernel invocations, finding optimal TTD configurations to improve training convergence. Our optimisations improve TTD-based GNN training throughput by up to 8.17x, reducing the GNN memory footprint by up to 5,474x without compromising model accuracy.

Degree

thesis:*
Name dc:type.qualificationname
Ph.D
Level dc:type.qualificationlevel
doctoral
Grantor dc:publisher.institution
University of Leeds
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Qiu, Shenghao
Advisors dc:contributor.advisor
  • Wang, Zheng
  • Xu, Jie
ORCID person:identifier.orcid

Rights

Language dc:language.dcterms_iso639_2
eng

Chain of custody

source
Harvested from
White Rose University Consortium
Base URL
etheses.whiterose.ac.uk/cgi/oai2
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
related terms
citation

Qiu, Shenghao. Accelerating Deep Learning by Optimising Communication and Computation Efficiency. doctoral thesis, University of Leeds, 2025.