University of Ioannina
Advanced reinforcement learning methods for autonomous navigation of unmanned surface vessels
Abstract
dc:descriptionThe increasing development of autonomous maritime systems has created a need for intelligent navigation methods capable of operating reliably in complex, dynamic, and uncertain environments. Unmanned Surface Vehicles (USVs) are particularly challenging autonomous platforms because their navigation is affected by environmental disturbances, moving vessels, limited perception, and the requirement to maintain safe separation from surrounding obstacles. While deep reinforcement learning (DRL) provides a promising framework for learning navigation policies through interaction with an environment, its application to autonomous maritime navigation remains constrained by issues related to safety, the availability of expert knowledge, generalization, and coordination among multiple autonomous vessels. This dissertation investigates advanced reinforcement learning methods for autonomous USV navigation, with particular emphasis on developing learning-based navigation policies that are effective, safe, and applicable to increasingly complex maritime scenarios.To establish a practical foundation for the research, a small-scale autonomous USV platform is designed and constructed to support the development, validation, and eventual transfer of learning-based navigation methods to physical hardware. In parallel, a comprehensive simulation framework is developed to enable efficient training and systematic evaluation without exposing the physical platform to the risks associated with trial-and-error learning. Within this framework, DRL-based navigation policies are investigated for autonomous USV navigation in environments containing environmental disturbances and moving maritime traffic. The resulting navigation framework demonstrates the ability of learning-based agents to integrate goal-directed navigation and collision avoidance within a continuous control setting. Building upon this foundation, imitation learning is investigated to address the difficulty of acquiring effective navigation policies solely through trial-and-error interaction. In particular, Generative Adversarial Imitation Learning (GAIL) is employed to derive navigation policies from expert demonstrations, enabling the agent to acquire desirable navigation behavior without explicitly reproducing a manually designed reward function. This provides an alternative mechanism for incorporating prior navigation knowledge into the learning process and demonstrates the applicability of adversarial imitation learning to autonomous USV navigation.The dissertation subsequently addresses safety as an explicit requirement of the learning process. Autonomous navigation is formulated as a Constrained Markov Decision Process (CMDP), in which task performance and safety are represented through separate reward and cost objectives. A Lagrangian relaxation framework is developed to incorporate constraints associated with static and moving obstacles into policy optimization. The approach enables the navigation policy to balance task performance with collision avoidance while explicitly accounting for different classes of maritime hazards. An extension of this framework further investigates trainable Lagrangian parameters and Actor-Critic-based optimization to provide adaptive safety enforcement during policy learning and the underlying interpretability of the safety constraints.Finally, the dissertation extends autonomous navigation from a single-USV setting to multi-USV environments. A Cross-Agent Multi-agent Soft Actor-Critic (CROSS-MASAC) framework is proposed in which decentralized actors are trained through cross-agent policy updates while leveraging shared learning information. The approach aims to improve coordination and policy consistency among independently acting USVs without requiring centralized execution. This establishes a transition from single-agent navigation toward scalable multi-agent maritime autonomy, where multiple vessels must simultaneously navigate and respond to one another within a shared environment.
Degree
thesis:*- Grantor dc:publisher
- University of Ioannina
- Year dc:date
- 2026
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Chaysri, Piyabhum
- Σάισι, Πιγάπουμ
Subjects
dc:subject × 35- Ενισχυτική μάθηση
- Βαθιά ενισχυτική μάθηση
- Ενισχυτική μάθηση με περιορισμούς
- Ασφαλής ενισχυτική μάθηση
- Πολυπρακτορικά συστήματα
- Πολλαπλασιαστές Lagrange με επίγνωση της κατάστασης
- Πλοήγηση μη επανδρωμένων σκαφών επιφανείας
- Αποφυγή συγκρούσεων
- Ερμηνεύσιμες πολιτικές
- Αυτόνομη πλοήγηση
- Μάθηση μέσω μίμησης
- Παραγωγικά ανταγωνιστικά δίκτυα
- Μεταφορά γνώσης
- Mηδενικής προσαρμογής
- Reinforcement learning
- Deep reinforcement learning (DRL)
- Safe reinforcement learning
- Constrained reinforcement Learning
- Mult-agent
- USV navigation
- Collision avoidance
- Interpretable policies
- State-aware Lagrange multipliers
- Imitation learning
- Generative adversarial networks
- Unmanned Surface Vessels
- Autonomous navigation
- Knowledge transfer
- Zero-shot learning
- Επιστήμη Ηλεκτρολόγου Μηχανικού, Ηλεκτρονικού Μηχανικού, Μηχανικού Η/Υ
- Επιστήμες Μηχανικού και Τεχνολογία
- Αυτοματισμοί και Συστήματα ελέγχου
- Electrical Engineering, Electronic Engineering, Information Engineering
- Engineering and Technology
- Automation and control systems
Rights
- Language dc:language
- eng
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/63028