Georgia Institute of Technology
Reinforcement learning with temporal logic and causal constraints
Abstract
dc:description.abstractReinforcement Learning (RL) represents a prominent machine learning paradigm that has been applied to a diverse array of decision-making challenges, including resource management, robot locomotion, recommendation systems, systems biology, traffic control, and achieving superhuman performance in video games. Nonetheless, RL's success has largely been confined to tightly controlled or constrained environments, and its effective application in many real-world contexts remains unrealized. One of the primary challenges underlying this limitation is the absence of a systematic, formalized approach to delineate requirements as tasks (or learning constraints), including those pertaining to causal and temporal ordering, and to develop methods that satisfy these constraints in a manner that is both sample-efficient and generalizable. This study aims to address these challenges by harnessing the capabilities of RL in learning high-performance controllers, while concurrently formulating the necessary assumptions and algorithms to ensure compliance with formal constraints. Specifically, we utilize Temporal Logic (TL) as a formal high-level language, enabling the straightforward specification of tasks comprised of temporally extended subgoals. The TL task, as defined by human operators, can be interpreted as a set of constraints for task fulfillment. Furthermore, additional environmental constraints, such as causal relationships, can be integrated into the TL task specifications through the use of deterministic finite automata (DFA). To address scenarios where these causal constraints are initially unknown, we propose an inductive logic programming (ILP) approach for their acquisition. Moreover, to mitigate issues related to unknown subgoal hierarchies and sparse rewards, we introduce a hierarchical RL framework that adopts a novel and simple critic shaping method for accelerating the constraint satisfaction. Then, it is of practical important to train a multi-task agent which generalize to solve an unseen TL task without further training. However, the issues of sub-task dependencies and sparse reward were not well addressed in the previous works related to generalization of TL task. Hence, in order to tack these issues, we present methodologies that leverage future-dependent options and implicit planning to enhance the zero-shot generalizability of RL agent. Finally, acknowledging that, in many practical applications, the mapping from environmental states to subgoals specified in TL may be unavailable—thereby limiting the applicability of extant RL algorithms—we propose a novel RL framework that derives subgoals through contrastive learning. The efficacy of the proposed frameworks is verified through a series of numerical experiments and benchmarks, which demonstrate their superiority relative to established methods.
Degree
thesis:*- Level thesis:degree_level
- Doctoral
- Department dc:contributor.department
- Electrical and Computer Engineering
- Grantor dc:publisher
- Georgia Institute of Technology
- Year dc:date.issued
- 2025
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Xu, Duo
- Advisor dc:contributor.advisor
-
- Fekri, Faramarz
- Committee members dc:contributor.committeemember
-
- Hale, Matthew
- Bansal, Suguman
- Sivakumar, Raghupathy
- Coogan, Samuel
Subjects
dc:subject × 4Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/81512