Democritus University of Thrace (DUTH)
Spiking neural networks for perception and control in human assistive robotic systems
Abstract
dc:descriptionThe growing demand for intelligent and energy-efficient systems in real-world applications has emerged the exploration of neuromorphic computing approaches, such as spiking neural networks (SNNs). The motivation behind this dissertation stems from the need to design computationally efficient, bioinspired learning models capable of handling complex perception and control tasks, particularly under constrained hardware and energy budgets. Spiking neural networks offer a promising path forward due to their temporal processing capabilities and compatibility with neuromorphic hardware. Thus, this thesis tackles a cascade problem in the context of multimodal perception and control by leveraging the unique properties of SNNs. Specifically, it focuses on the development and evaluation of spike-based learning architectures across three interrelated domains: discriminative multimodal feature discrimination, object manipulation and robotic arm control via reinforcement learning (RL). An important application of this work is Ambient Assisted Living (AAL), where systems must operate reliably and efficiently in non-intrusive ways to support the well-being and autonomy of elderly individuals. The contribution of this dissertation to the state-of-the-art is substantiated through a series of peer-reviewed publications. Additionally the main contributions can be summarized as follows: Introduces the first spiking feature discrimination framework, opening a novel research direction in multimodal spiking perception. Presents the first successful application of gradient-based RL with spiking neural networks for robotic control. Demonstrates the application of this approach to complex tasks such as object manipulation, showcasing its practical utility in robotic systems. Advances biologically inspired machine intelligence for human-assistive robotic systems, particularly within the scope of AAL. The dissertation at hand is articulated into six chapters. Beginning with Chapter 1, it laid the theoretical foundation of this dissertation. It introduces the principles of spiking neural networks, neuron models (e.g., LIF, Izhikevich), encoding strategies (i.e., rate, temporal), and plasticity mechanisms. Additionally, this chapter presents an extended literature review, which presents spiking RL approaches in robotics systems, focused on learning mechanisms, neuronal models, and evaluation metrics. The spiking RL models fall into two primary categories, the bioinspired (e.g. STDP-based) and the gradient-based adaptations of traditional RL methods. Thus, we further categorized the surveyed works based in robotic application, extracting valuable information about inadequacies in the field along with future directions. This chapter also provides an overview of feature discrimination methods and multimodal fusion approaches. Lastly, it clarifies the baseline of SNN research and contextualizes the challenges related to convergence, scalability, and real-world deployment. Chapter 2 presents the databases exploited to conduct our empirical studies. The above includes the utilized datasets from the field of audio-visual classification, along with the custom one, specifically collected for the task of elderly human action recognition in the field of AAL. The chapter also introduces the metrics adopted to quantify classification performance, feature separability, and robotic control accuracy. Proceeding with the main analysis of our study, Chapter 3 introduces a novel spiking multimodal fusion architecture for perception tasks. It paves the path in spiking perception with a spiking multimodal feature discrimination pipeline, where both visual and auditory modalities are processed using SNNs. Unlike existing multimodal approaches using traditional deep layers, the proposed pipeline emphasizes early-stage feature discrimination using spiking layers tailored for each modality. A spiking residual network processes the visual input, while an efficient auditory SNN complements the temporal dynamics of the auditory inputs. Subsequently, these networks are fused through a novel spiking MLP (SMLP) incorporating an L2-normalization layer to maximize angular separation in the latent feature space. The results reveal that spiking feature discrimination is a prominent research field, not only matching but in some approaches exceeding the performance of classical networks under low-power constraints. Following this, an action recognition application for AAL is examined. Here, the challenge is to identify and classifyelderly individuals’ actions using non-intrusive, low-power sensor setups. This chapter introduces a complete pipeline for real-time action recognition based on RGB video inputs. The proposed approach utilizes a biologically-inspired Hierarchical Temporal Memory (HTM), and SNNs implemented in Nengo and evaluates their suitability in constrained data availability approaches. The results reveal that spiking models can not only match but also outperform conventional models. This application directly supports the integration of bioinspired intelligence in AAL systems that must function continuously and discreetly to support human living. The fourth and fifth chapters focus on the integration of spiking neural networks into reinforcement learning frameworks for robotic manipulation tasks. Recognizing the limitations of conventional deep reinforcement learning (DRL) approaches, such as their high energy consumption and dependence on resource-intensive training, these chapters proposes a hybrid framework that combines the temporal efficiency of SNNs with the optimization capacity of deep networks. Specifically, a hybrid Deep Deterministic Policy Gradient (DDPG) algorithm is introduced, where the actor network is realized as a spiking neural network while the critic is retained as a conventional deep network, used solely during the training phase. This architectural decoupling facilitates several methodological advantages. First, the spiking actor network is trained to generate motor commands directly from sensory states, enabling real-time inference compatible with deployment. Once training is complete, the energy-intensive critic is discarded, and only the spiking actor remains active during inference, significantly reducing computational and power demands. This design was specifically designed and adapted for two distinct robotic control scenarios. The first involves a 2-degree-of-freedom robotic arm equipped with a dual-finger gripper, tasked with performing object manipulation through reach, grasp, and transfer operations. The second scenario addresses a more complex 6-degree-of-freedom robotic arm required to perform 3D target-reaching, incorporating both RGB imagery and laser-based range sensing for planning and obstacle avoidance. Experimental results across both scenarios demonstrated that the hybrid approach not only matches but in some cases surpasses the performance of classical DDPG baselines in terms of task success rate, trajectory smoothness, and convergence time. More importantly, it offers a substantial improvement in energy efficiency, as the spiking actor alone is responsible for action generation during deployment. This validates the practical viability of the proposed hybrid architecture in scenarios where energy constraints are present, such as in AAL systems. The dissertation concludes with Chapter 6 summarizing our findings and their contribution to the field. Finally, we discuss the open challenges and future directions that aim to extend the research described here.
Degree
thesis:*- Grantor dc:publisher
- Democritus University of Thrace (DUTH)
- Year dc:date
- 2025
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Oikonomou, Aikaterini-Maria
- Οικονόμου, Αικατερίνη-Μαρία
Subjects
dc:subject × 21- Νευρωνικά δίκτυα αιχμής
- Οικονόμου, Αικατερίνη Μαρία
- Δημοκρίτειο Πανεπιστήμιο Θράκης
- Οικονόμου
- Νευρωνικά δίκτυα αιχμών για αντίληψη και έλεγχο σε ρομποτικά συστήματα υποβοήθησης ανθρώπου
- Ξάνθη
- 2025
- Spiking neural networks
- Oikonomou, Aikaterini Maria
- Democritus University of Thrace
- Oikonomou
- Spiking neural networks for perception and control in human assistive robotic systems
- Xanthi
- Επιστήμη Ηλεκτρολόγου Μηχανικού, Ηλεκτρονικού Μηχανικού, Μηχανικού Η/Υ
- Επιστήμες Μηχανικού και Τεχνολογία
- Ρομποτική
- Αυτοματισμοί και Συστήματα ελέγχου
- Electrical Engineering, Electronic Engineering, Information Engineering
- Engineering and Technology
- Robotics
- Automation and control systems
Rights
- Language dc:language
- eng
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/59503