Back to search

Cornell University

Adaptive Allocation of Compute Capacity in Large Language Models

Abstract

dc:description.abstract

Large language models apply compute uniformly by default, yet structural asymmetries at every stage of the model lifecycle make this uniformity systematically wasteful. This thesis identifies three such asymmetries and proposes a targeted solution to each. At the training stage, reinforcement learning post-training is dominated by rollout generation cost, yet existing methods allocate rollout budget uniformly across prompts regardless of their informativeness. Pilot-Commit addresses this with a two-stage allocation framework that estimates per-prompt reward variance with a small pilot budget and concentrates the remaining rollouts on high-variance prompts, matching baseline accuracy with significantly fewer total rollouts. At the inference stage, the prefill and decode phases have mismatched computational profiles: prefill is compute-bound while decode is memory-bound. OverFill exploits this by using the full model for prefill and a width-pruned model for decode, achieving Pareto-optimal accuracy-latency tradeoffs across multiple model scales while incurring minimal latency overhead. At the deployment stage, developers rely on expensive large-model API calls to implement fuzzy functions that resist symbolic specification, paying the cost of intent encoding at every invocation. Program-as-Weights introduces a compiler-interpreter paradigm in which a large neural compiler translates specifications into compact neural programs that a small fixed interpreter executes locally, amortizing compilation cost across all future invocations. Across training, inference, and deployment, the same principle holds: identifying where compute produces value and reallocating accordingly yields substantial gains without changing the underlying objectives.

Degree

thesis:*
Name thesis:degree_name
Ph. D., Computer Science
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Computer Science
Grantor
Cornell University
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Kim, Woojeong
Chair dc:contributor.chair
  • Rush, Alexander
Committee members dc:contributor.committeemember
  • Abdelfattah, Mohamed
  • Kuleshov, Volodymyr

Subjects

dc:subject × 1

Rights

Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Other
Kim_cornellgrad_0058F_15523
https://www.proquest.com/LegacyDocView/DISSNUM/32673301

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Kim, Woojeong. Adaptive Allocation of Compute Capacity in Large Language Models. Doctor of Philosophy thesis, Cornell University, 2026. https://hdl.handle.net/1813/126551