Back to search

University of Glasgow

Interactive shared visuo-tactile perception and learning in robotics

Abstract

dc:description.abstract

Humans possess the ability to seamlessly integrate perceptual information from vision and tactile sensing to maintain a high-level cognitive understanding of the environment. Similarly, leveraging vision and tactile sensing can enable robots to interact with novel objects in unstructured environments. This thesis presents novel approaches for visuo-tactile perception and learning in robotics, focussing on object pose estimation, recognition, and reconstruction. For robust pose estimation of unknown objects in dense clutter, a novel recursive filtering formulation termed translation-invariant Quaternion filter (TIQF) and its global-optimal version stochastic TIQF (S-TIQF) with visuo-tactile point cloud data are proposed in this thesis. A two-robot team with vision and tactile sensing autonomously declutters the scene and retrieves the pose of the target object which can be opaque or transparent using STIQF. Moreover, S-TIQF is deployed to correct hand-eye calibration with arbitrary objects in-situ which is necessary for shared perception. In addition to rigid objects, the pose tracking of articulated objects is a challenging task that requires integration of vision and tactile sensing. A novel Manifold Unscented Kalman Filter method on the SE(3) Lie Group termed ArtReg is presented, which is used for tracking the pose of objects. Using ArtReg, a novel framework is designed for detecting, tracking, and the goal-driven manipulation of unknown objects (single, multiple, or articulated) without assuming any prior knowledge regarding object shape or dynamics. This thesis also presents a new vision-to-tactile cross-modal learning approach for object recognition where the network is trained with dense visual point clouds and tested with sparse point clouds acquired from tactile sensors. A novel unsupervised domain adaptation loss function is proposed to minimise the gap between the visual and tactile domains. Cross-modal adaptation allows the robotic system to switch to tactile sensing in case vision sensing is compromised, thereby increasing the robustness of the system. Object reconstruction is another fundamental perceptual challenge that enables downstream tasks such as pose estimation and recognition. This thesis introduces a novel deep learning-based 3D object reconstruction approach utilising sparse tactile point cloud data to accurately recover the geometry of category-level unknown transparent objects leveraging only synthetic data for training. Furthermore, it is also demonstrated with visuo-tactile point clouds for opaque objects, wherein the tactile data are used to refine the shape in regions of uncertainty in the visual data. The proposed methods have been rigorously validated with extensive experiments on standard datasets and robot experiments and have been demonstrated to outperform state of-the-art approaches.

Degree

thesis:*
Level dc:type.qualificationlevel
PhD
Grantor dc:publisher.institution
University of Glasgow
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Murali, Prajval Kumar

Subjects

dc:subject × 1

Rights

Language dc:language
en

Identifiers

dc:identifier.*

Chain of custody

source
Harvested from
University of Glasgow
Base URL
theses.gla.ac.uk/cgi/oai2
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Murali, Prajval Kumar. Interactive shared visuo-tactile perception and learning in robotics. PhD thesis, University of Glasgow, 2025. https://doi.org/10.5525/gla.thesis.85001