Georgia Institute of Technology
Provably Robust Offline Meta-Reinforcement Learning under Bagged Rewards via Spectral Representations
Abstract
dc:description.abstractMeta-reinforcement learning trains agents to adapt to new tasks by drawing on experience from related tasks. In the offline setting, the agent must learn entirely from a fixed dataset without further interaction with the environment. A common approach, known as context-based offline meta-reinforcement learning (COMRL), infers a latent task representation from the agent's recent experience and uses it to guide the policy. Existing COMRL methods rely on dense reward signals, where the agent receives feedback after every action. However, in many real-world settings, rewards are not observed at every step. Instead, they are aggregated into 'bags' that summarize feedback over a sequence of actions. In this thesis, we prove that under such bagged reward schedules, the task information available at each step vanishes, causing current methods to fail. To address this problem, we propose SpectralMeta, an algorithm that separates the learning of environment dynamics from task inference. The algorithm first pre-trains a spectral representation of the environment using reward-free transitions. It then uses this representation to cast task inference as a closed-form Bayesian linear regression over the sparse aggregated rewards. This approach does not require per-step reward signals to identify the task. We provide theoretical guarantees on reward recovery and policy quality, and identify a fundamental limit: the amount of context needed grows quadratically with reward sparsity. In experiments on continuous-control benchmarks, SpectralMeta maintains strong performance even when rewards are reduced to a single scalar per episode; a setting where all existing baselines collapse to behavior cloning.
Degree
thesis:*- Name thesis:degree_name
- Computer Science, MS
- Department thesis:degree_department
- Computational Science and Engineering
- Grantor
- Georgia Institute of Technology
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Vaidya, Yashas
- Advisor dc:contributor.advisor
-
- Dai, Bo
- Committee member dc:contributor.committeemember
-
- Pan Li
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/81616