Back to search

Georgia Institute of Technology

Reinforcement learning with temporal logic and causal constraints

Abstract

dc:description.abstract

Reinforcement Learning (RL) represents a prominent machine learning paradigm that has been applied to a diverse array of decision-making challenges, including resource management, robot locomotion, recommendation systems, systems biology, traffic control, and achieving superhuman performance in video games. Nonetheless, RL's success has largely been confined to tightly controlled or constrained environments, and its effective application in many real-world contexts remains unrealized. One of the primary challenges underlying this limitation is the absence of a systematic, formalized approach to delineate requirements as tasks (or learning constraints), including those pertaining to causal and temporal ordering, and to develop methods that satisfy these constraints in a manner that is both sample-efficient and generalizable. This study aims to address these challenges by harnessing the capabilities of RL in learning high-performance controllers, while concurrently formulating the necessary assumptions and algorithms to ensure compliance with formal constraints. Specifically, we utilize Temporal Logic (TL) as a formal high-level language, enabling the straightforward specification of tasks comprised of temporally extended subgoals. The TL task, as defined by human operators, can be interpreted as a set of constraints for task fulfillment. Furthermore, additional environmental constraints, such as causal relationships, can be integrated into the TL task specifications through the use of deterministic finite automata (DFA). To address scenarios where these causal constraints are initially unknown, we propose an inductive logic programming (ILP) approach for their acquisition. Moreover, to mitigate issues related to unknown subgoal hierarchies and sparse rewards, we introduce a hierarchical RL framework that adopts a novel and simple critic shaping method for accelerating the constraint satisfaction. Then, it is of practical important to train a multi-task agent which generalize to solve an unseen TL task without further training. However, the issues of sub-task dependencies and sparse reward were not well addressed in the previous works related to generalization of TL task. Hence, in order to tack these issues, we present methodologies that leverage future-dependent options and implicit planning to enhance the zero-shot generalizability of RL agent. Finally, acknowledging that, in many practical applications, the mapping from environmental states to subgoals specified in TL may be unavailable—thereby limiting the applicability of extant RL algorithms—we propose a novel RL framework that derives subgoals through contrastive learning. The efficacy of the proposed frameworks is verified through a series of numerical experiments and benchmarks, which demonstrate their superiority relative to established methods.

Degree

thesis:*
Level thesis:degree_level
Doctoral
Department dc:contributor.department
Electrical and Computer Engineering
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Xu, Duo
Advisor dc:contributor.advisor
  • Fekri, Faramarz
Committee members dc:contributor.committeemember
  • Hale, Matthew
  • Bansal, Suguman
  • Sivakumar, Raghupathy
  • Coogan, Samuel

Subjects

dc:subject × 4

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/81512

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Xu, Duo. Reinforcement learning with temporal logic and causal constraints. Doctoral thesis, Georgia Institute of Technology, 2025. https://hdl.handle.net/1853/81512