{"id":{"repo_id":"etsu","oai_identifier":"oai:dc.etsu.edu:etd-6235"},"canonical_url":"https://search.ndltd.org/etd/etsu/oai:dc.etsu.edu:etd-6235","repository":{"repo_id":"etsu","name":"East Tennessee State University","base_url":"https://dc.etsu.edu/do/oai/"},"display":{"title":"Durable, Distributed LLM Inference on COTS Devices","abstract":"The advancement of Large Language Models (LLMs) has fundamentally changed the nature of natural language processing. The substantial memory requirements of frontier models creates a significant barrier to entry, centralizing inference. This thesis presents the design and implementation of a distributed inference framework designed to democratize LLMs by leveraging commodity devices. The framework combines the resources of heterogeneous COTS devices into a unified compute pool, enabling the inference of models exceeding a single device's memory capacity. A novel Task Partitioning Engine (TPE) analyzes model architectures, profiles node capabilities, and supports pipeline and expert parallelism strategies. The primary contribution is a fault tolerance mechanism which detects node failures during active inference via heartbeat monitoring, automatically recovers lost model shards, and resumes generation from the point of failure with zero token loss. Evaluation on a heterogeneous cluster demonstrates successful distributed inference across heterogeneous devices and validates mid-inference recovery following node failure.","abstract_html":"The advancement of Large Language Models (LLMs) has fundamentally changed the nature of natural language processing. The substantial memory requirements of frontier models creates a significant barrier to entry, centralizing inference. This thesis presents the design and implementation of a distributed inference framework designed to democratize LLMs by leveraging commodity devices. The framework combines the resources of heterogeneous COTS devices into a unified compute pool, enabling the inference of models exceeding a single device&#x27;s memory capacity. A novel Task Partitioning Engine (TPE) analyzes model architectures, profiles node capabilities, and supports pipeline and expert parallelism strategies. The primary contribution is a fault tolerance mechanism which detects node failures during active inference via heartbeat monitoring, automatically recovers lost model shards, and resumes generation from the point of failure with zero token loss. Evaluation on a heterogeneous cluster demonstrates successful distributed inference across heterogeneous devices and validates mid-inference recovery following node failure.","abstract_has_math":false,"creators":["Dunn, Brycen E"],"institution":"East Tennessee State University","degree_name":"MS (Master of Science)","degree_level":"Thesis - unrestricted","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":[],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2026,"date_issued":"2026-05-01T07:00:00Z","date_published":"2026-05-01T07:00:00Z","updated_at":"2026-11-19T07:57:30Z","subjects":["networking","deep learning","distributed inference","edge computing","Systems Architecture"],"languages":["eng"],"rights":["Copyright by the authors."],"rights_urls":[],"identifier_entries":[],"person_identifier_entries":[]},"links":{"outbound_url":"https://dc.etsu.edu/etd/4666","outbound_label":"Repository record","outbound_source":"dc:identifier","record_url":"https://dc.etsu.edu/etd/4666","record_label":"Repository record","record_kind":"record","record_source":"dc:identifier","download_url":"https://dc.etsu.edu/context/etd/article/6235/viewcontent/DunnB041926f.pdf","download_label":"Download PDF","download_source":"dc:identifier","access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://dc.etsu.edu/do/oai/?verb=GetRecord&metadataPrefix=oai_etdms&identifier=oai%3Adc.etsu.edu%3Aetd-6235","prefix":"oai_etdms"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:creator","label":"Author","values":["Dunn, Brycen E"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.available","label":"Dc Date Available","values":["2026-04-05T07:00:00Z"]},{"key":"dc:date.issued","label":"Date","values":["2026-05-01T07:00:00Z"]},{"key":"dc:publisher","label":"Institution","values":["East Tennessee State University"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis - unrestricted"]},{"key":"thesis:degree_name","label":"Degree Name","values":["MS (Master of Science)"]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["East Tennessee State University"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["networking","deep learning","distributed inference","edge computing","Systems Architecture"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright by the authors."]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://dc.etsu.edu/context/etd/article/6235/viewcontent/DunnB041926f.pdf","https://dc.etsu.edu/etd/4666"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["The advancement of Large Language Models (LLMs) has fundamentally changed the nature of natural language processing. The substantial memory requirements of frontier models creates a significant barrier to entry, centralizing inference. This thesis presents the design and implementation of a distributed inference framework designed to democratize LLMs by leveraging commodity devices. The framework combines the resources of heterogeneous COTS devices into a unified compute pool, enabling the inference of models exceeding a single device's memory capacity. A novel Task Partitioning Engine (TPE) analyzes model architectures, profiles node capabilities, and supports pipeline and expert parallelism strategies. The primary contribution is a fault tolerance mechanism which detects node failures during active inference via heartbeat monitoring, automatically recovers lost model shards, and resumes generation from the point of failure with zero token loss. Evaluation on a heterogeneous cluster demonstrates successful distributed inference across heterogeneous devices and validates mid-inference recovery following node failure."]},{"key":"dc:title","label":"Title","values":["Durable, Distributed LLM Inference on COTS Devices"]}]}],"canonical_facts":{"dc:creator":["Dunn, Brycen E"],"dc:date.available":["2026-04-05T07:00:00Z"],"dc:date.issued":["2026-05-01T07:00:00Z"],"dc:description.abstract":["The advancement of Large Language Models (LLMs) has fundamentally changed the nature of natural language processing. The substantial memory requirements of frontier models creates a significant barrier to entry, centralizing inference. This thesis presents the design and implementation of a distributed inference framework designed to democratize LLMs by leveraging commodity devices. The framework combines the resources of heterogeneous COTS devices into a unified compute pool, enabling the inference of models exceeding a single device's memory capacity. A novel Task Partitioning Engine (TPE) analyzes model architectures, profiles node capabilities, and supports pipeline and expert parallelism strategies. The primary contribution is a fault tolerance mechanism which detects node failures during active inference via heartbeat monitoring, automatically recovers lost model shards, and resumes generation from the point of failure with zero token loss. Evaluation on a heterogeneous cluster demonstrates successful distributed inference across heterogeneous devices and validates mid-inference recovery following node failure."],"dc:identifier":["https://dc.etsu.edu/context/etd/article/6235/viewcontent/DunnB041926f.pdf","https://dc.etsu.edu/etd/4666"],"dc:language":["eng"],"dc:publisher":["East Tennessee State University"],"dc:rights":["Copyright by the authors."],"dc:subject":["networking","deep learning","distributed inference","edge computing","Systems Architecture"],"dc:title":["Durable, Distributed LLM Inference on COTS Devices"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis - unrestricted"],"thesis:degree_name":["MS (Master of Science)"],"thesis:institution_name":["East Tennessee State University"]},"updated_at":"2026-11-19T07:57:30Z"}