Back to search

East Tennessee State University

Durable, Distributed LLM Inference on COTS Devices

Abstract

dc:description.abstract

The advancement of Large Language Models (LLMs) has fundamentally changed the nature of natural language processing. The substantial memory requirements of frontier models creates a significant barrier to entry, centralizing inference. This thesis presents the design and implementation of a distributed inference framework designed to democratize LLMs by leveraging commodity devices. The framework combines the resources of heterogeneous COTS devices into a unified compute pool, enabling the inference of models exceeding a single device's memory capacity. A novel Task Partitioning Engine (TPE) analyzes model architectures, profiles node capabilities, and supports pipeline and expert parallelism strategies. The primary contribution is a fault tolerance mechanism which detects node failures during active inference via heartbeat monitoring, automatically recovers lost model shards, and resumes generation from the point of failure with zero token loss. Evaluation on a heterogeneous cluster demonstrates successful distributed inference across heterogeneous devices and validates mid-inference recovery following node failure.

Degree

thesis:*
Name thesis:degree_name
MS (Master of Science)
Level thesis:degree_level
Thesis - unrestricted
Discipline thesis:degree_discipline
Computer Science
Grantor dc:publisher
East Tennessee State University
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Dunn, Brycen E

Subjects

dc:subject × 5

Rights

dc:rights
Statement dc:rights
  • Copyright by the authors.
Language dc:language
eng

Identifiers

dc:identifier.*
Repository record dc:identifier
https://dc.etsu.edu/etd/4666

Chain of custody

source
Harvested from
East Tennessee State University
Base URL
dc.etsu.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Dunn, Brycen E. Durable, Distributed LLM Inference on COTS Devices. Thesis - unrestricted thesis, East Tennessee State University, 2026. https://dc.etsu.edu/etd/4666