Back to search

Cornell University

Reinforcement Learning with Latent Low Rank Structure

Abstract

dc:description.abstract

The practicality of reinforcement learning algorithms has been limited due to poor scaling with respect to the problem size; without structural assumptions, algorithms incur a sample complexity bound or regret bound that scales with the product of the sizes of the state space $|S|$ and action space $|A|$ of a finite horizon MDP, e.g., $|S||A|$ or $\sqrt{|S||A|}$. In this work, we consider a class of MDPs that have latent low rank structure. We first investigate what are the minimal low rank assumptions that admit efficient reinforcement learning. We first show that without imposing further assumptions beyond low rank of Q*, if one is constrained to estimate the $Q$ function using only observations from a subset of entries, there is a worst case instance in which one must incur a sample complexity exponential in the horizon $H$ to learn a near optimal policy. We subsequently show that under stronger low rank structural assumptions, given access to a generative model, Low Rank Monte Carlo Policy Iteration (LR-MCPI) and Low Rank Empirical Value Iteration (LR-EVI) achieve the desired sample complexity of \tilde{O}\left((|S|+|A|)poly(d,H)/ε2\right) for a rank $d$ setting, which is minimax optimal with respect to the scaling of $|S|, |A|$, and ε. In contrast to literature on linear and low-rank MDPs, we do not require a known feature mapping, our algorithm is computationally simple, and our results hold for long time horizons. We next consider the problem of transferring a latent low rank representation when the source and target MDPs have transition kernels with Tucker rank $(|S|, d, |A| )$, $(|S| , |S| , d), (d, |S| , |A| )$, or $(d , d , d )$. In each setting, we introduce the transfer-ability coefficient α that measures the difficulty of representational transfer. Our algorithm learns latent representations in each source MDP and then exploits the linear structure to remove the dependence on $|S| , |A| $, or $|S| |A| $ in the target MDP regret bound. We complement our positive results with information theoretic lower bounds that show our algorithms (excluding the ($d, d, d$) setting) are minimax-optimal with respect to α.

Degree

thesis:*
Name thesis:degree_name
Ph. D., Operations Research and Information Engineering
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Operations Research and Information Engineering
Grantor
Cornell University
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Sam, Tyler
Chair dc:contributor.chair
  • Yu, Christina
Committee members dc:contributor.committeemember
  • Chen, Yudong
  • Sun, Wen
  • Frazier, Peter

Subjects

dc:subject × 1

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Bibid
17422616
Dc Identifier Other
ProQuest Submission ID: 15218
ProQuest Publication ID: 32170678

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Sam, Tyler. Reinforcement Learning with Latent Low Rank Structure. Doctor of Philosophy thesis, Cornell University, 2025. https://hdl.handle.net/1813/120914