Back to search

Technical University of Crete (TUC)

Deep implicit imitation reinforcement learning

Abstract

dc:description

Imitation learning (IL) enables agents to acquire complex behaviours by learning from expert demonstrations, offering a compelling alternative to pure trial-and-error reinforcement learning. However, conventional imitation learning approaches face critical practical limitations that restrict their real-world applicability: the requirement for complete state-action demonstrations with explicit action labels; the assumption that expert demonstrations represent optimal behaviour; and the assumption of homogeneous expert and trainee agent action spaces. These constraints create substantial barriers in scenarios where action recording is technically infeasible, privacy-sensitive, or prohibitively expensive; where the expert is known to be suboptimal, or optimal performance is undefined or potentially unattainable; and where an expert agent and a trainee agent possess fundamentally different action capabilities.Against this background, this thesis puts forward a novel deep implicit imitation reinforcement learning framework that fundamentally addresses these limitations. Specifically, our framework enables agents to learn from state observations-only, suboptimal expert datasets, while possessing the capacity to surpass expert performance through continuous environmental interaction.Our framework gives rise to two novel deep reinforcement learning (DRL) algorithms: Deep Implicit Imitation Q-Network (DIIQN) for homogeneous action spaces, and Heterogeneous Actions DIIQN (HA-DIIQN) for scenarios where expert and agent possess fundamentally different action capabilities. DIIQN extends the classic Deep Q-Network (DQN) algorithm with mechanisms for reconstructing expert actions from observations, identifying relevant expert demonstrations, and dynamically weighing expert guidance against self-directed learning throughout training.HA-DIIQN builds on DIIQN to tackle heterogeneous action settings, in which expert and agent possess different action sets—that is, to address scenarios where direct replication of expert demonstrations is impossible. Employing infeasibility identification and so-called “bridge discovery” procedures that we introduce, HA-DIIQN manages to identify alternative feasible pathways that effectively align agent capabilities to expert state trajectories. Thus, HA-DIIQN enables knowledge transfer across agents with different control schemes, allowing them to cope in situations in which conventional imitation learning methods fail entirely. A key component of our framework is a novel confidence mechanism that is used by both our algorithms to adaptively determine when to follow expert guidance and when to rely on the agent’s own superior discoveries, enabling the framework to accelerate initial learning while preserving autonomous improvement beyond expert limitations. In a nutshell, our main contributions in this thesis are the following: (a) we put forward the first model-free DRL framework for implicit imitation learning; (b) we employ this framework to substantially accelerate the training of DRL methods; (c) we introduce two novel DRL algorithms that enable implicit imitation learning from suboptimal experts with the ability to surpass them; (d) interestingly, for the first time in the literature, one of our algorithms allows for knowledge transfer between and expert and a trainee agent that possess different action sets—i.e., operate in heterogeneous action spaces; and (e) we demonstrate our framework’s effectiveness in both classic DRL environments, and in the challenging autonomous driving in lane-free traffic domain. We note that ours is effectively the first deep reinforcement learning framework for implicit imitation learning.Beyond methodological innovation, this thesis also contributes a novel taxonomy of imitation learning methods that organizes the field into three overarching paradigms: explicit imitation, implicit imitation, and inverse reinforcement learning. This taxonomy offers a clear and structured view of how different IL approaches, and in particular modern IL approaches that have appeared in the past decade, relate to one another. We conduct a comprehensive experimental evaluation of our algorithms across diverse environments spanning game-playing (MinAtar suite), navigation (2D Maze, Point Maze), and autonomous driving (lane-free traffic). Our results indicate that DIIQN demonstrates up to 136% improvement in episodic returns, and up to 67% reduction in training time compared to standard Deep Q-Network (DQN), while consistently surpassing in terms of reward-gathering performance established implicit imitation learning methods (BCO, GAIfO, and ORIL). The latter are shown to not be able to exceed the performance of suboptimal experts, in contrast to our method which does so consistently. At the same time, when operating in heterogeneous action settings, our results demonstrate that HA-DIIQN agents are able to successfully leverage (heterogeneous) expert datasets that are by and large of no worth to conventional imitation learning approaches. In particular, HA-DIIQN is shown to achieve up to 64% faster convergence to the optimal policy compared to standard DQN, and up to 52% faster convergence relative to standard DIIQN—across scenarios with varying degrees of action space heterogeneity, ranging from partial action sets overlap to exclusive actions sets disjunction. Moreover, we conduct a thorough parameter sensitivity analysis that validates the stability and robustness of our approach. In particular, our analysis confirms that our dynamic confidence mechanism substantially reduces training variance Additionally, our analysis allows us to provide specific recommendations for appropriate hyperparameter configurations (regarding different expert data sizes, confidence thresholds, and thresholds determining the required similarity between states in the expert and trainee agent trajectories). The practical implications of our PhD work are substantial. On the one hand, our framework does not require expert action labeling, unlike most traditional (“explicit’) imitation learning methods. This contributes to the “viability” of imitation learning as a learning paradigm for scenarios involving imprecise or error-prone human observation, or for applications that require the adherence to privacy requirements or to resource-limited budgets. In addition, our framework offers the ability to learn from and surpass suboptimal experts, via leveraging readily available demonstrations from competent but imperfect demonstrators rather than requiring (the potentially expensive) verification by optimal experts. Moreover, the ability of our framework to tackle heterogeneous action spaces, creates the potential for cross-platform knowledge transfer across distinct hardware generations; for simulation-to-reality transitions; and for evolving system capabilities without requiring new dataset collection. Finally, the training time reductions demonstrated in our results, indicate the potential for reduced computational costs, faster deployment cycles, and improved feasibility for real-world applications that cannot afford the luxury of lengthy (or potentially dangerous) exploration processes. In the end, the most valuable expert is not the optimal one, but the accessible one; and our novel deep implicit imitation reinforcement learning framework finally makes that expertise accessible and thus worth distilling from.

Degree

thesis:*
Grantor dc:publisher
Technical University of Crete (TUC)
Year dc:date
2025

Author and committee

dc:creator, dc:contributor.*
Authors dc:creator
  • Chrysomallis, Iason
  • Χρυσομάλλης, Ιάσων

Subjects

dc:subject × 12

Rights

Language dc:language
eng

Identifiers

dc:identifier.*
Identifier
10.12681/eadd/60483

Chain of custody

source
Harvested from
Greek National Archive of PhD Theses
Base URL
phdtheses.ekt.gr/eadd_oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Chrysomallis, Iason; Χρυσομάλλης, Ιάσων. Deep implicit imitation reinforcement learning. Technical University of Crete (TUC), 2025. http://hdl.handle.net/10442/hedi/60483