Boston University
Finite-sample analysis for distributed and neural reinforcement learning
Abstract
dc:description.abstractThis dissertation develops a modular finite-sample theory for reinforcement learning under two realities of modern practice that challenge classical analysis: nonlinear function approximation and trajectory-based, temporally dependent data, including distributed and continuing settings. The central objective is to provide reproducible end-to-end convergence guarantees that explicitly separate the roles of approximation error, Markov dependence, and algorithm design. The theory is organized around four tightly connected components—policy evaluation, policy improvement, distributed learning, and continuous learning—with bounds that remain interpretable across regimes. For policy evaluation, the dissertation analyzes neural temporal-difference learning through a gradient-splitting viewpoint that decomposes the mean update into a dominant descent term plus a curvature remainder, yielding Lyapunov recursions that cleanly isolate descent, stochastic error under Markov sampling, and an approximation floor. For policy improvement, it studies neural actor–critic methods and derives finite-sample stationarity guarantees by combining actor progress with critic tracking and closing the resulting feedback loop via a small-gain argument, making the required balance among stepsizes, approximation, and mixing explicit. For distributed learning, it establishes finite-sample guarantees for TD(λ) with one-shot averaging and identifies regimes with linear speedup in the dominant error term while remaining communication-efficient under Markov sampling. For continuous learning, it develops and analyzes average-reward TD-style algorithms that enforce a steady-state normalization using either a double-chain construction or a single-chain auxiliary recursion, proving finite-sample convergence rates that expose dependence on stepsizes, feature conditioning, and mixing time.
Degree
thesis:*- Name thesis:degree_name
- Doctor of Philosophy
- Level thesis:degree_level
- doctoral
- Discipline thesis:degree_discipline
- Electrical & Computer Engineering
- Grantor
- Boston University
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Tian, Haoxing
- Advisors dc:contributor.advisor
-
- Paschalidis, Ioannis Ch.
- Olshevsky, Alexander
- ORCID dc:identifier.orcid
Subjects
dc:subject × 1Rights
dc:rights- Statement dc:rights
-
- Attribution-NonCommercial-NoDerivatives 4.0 International
- Licence dc:rights.uri
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/2144/53020