Back to search

University of Nevada, Reno

Enhancing Human-Robot Collaboration Through a Multi-Module Interaction Framework With Sensor Fusion: Object Recognition, Verbal Communication, User(s) of Interest Detection, Gesture and Gaze Recognition

Abstract

dc:description.abstract

With the increasing presence of robots in our daily lives, it is crucial to design interaction interfaces that are natural, easy to use, and meaningful for a wide range of robotic tasks. This is important not only to enhance the user experience but also to increase task reliability by providing supplementary, task-specific contextual information if needed. Motivated by these goals, we propose a multi-modal framework consisting of multiple independent modules. These modules take advantage of multiple sensors (e.g. image, sound, depth) and can be used separately or in combination for effective human-robot collaborative interaction. We identified and implemented four key components of an effective human-robot collaborative setting, which include: (1) determining Object(s) of Interest location and pose, (2) extracting intricate information from verbal instructions, (3) resolving User(s) of Interest (UOI), and (4) providing gesture recognition and gaze estimation to facilitate natural and intuitive interactions. The system uses a feature-detector-descriptor approach for object recognition and a homography-based technique for planar pose estimation and a deep multi-task learning model to extract intricate task parameters from verbal communication. The User(s) of Interest (UOI) is detected by estimating facing state and active speakers. The framework also includes gesture detection and gaze estimation modules, which are combined with verbal instruction components to form structured commands for robotic entities. Experiments were conducted to assess the performance of these interaction interfaces, and the results demonstrated the effectiveness of the proposed approach.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy
Level thesis:degree_level
Doctorate Degree
Department thesis:degree_department
Computer Science
Grantor
University of Nevada, Reno
Year dc:date.issued
2024

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Paul, Shuvo Kumar
Advisor dc:contributor.advisor
  • Nicolescu, Mircea
Committee members dc:contributor.committeemember
  • Nicolescu, Monica
  • Tavakkoli, Alireza
  • Dascalu, Sergiu
  • Deaconu, Valentin

Subjects

dc:subject × 6

Rights

dc:rights
Statement dc:rights
  • Creative Commons Attribution 4.0 International
  • Author(s)

Identifiers

dc:identifier.*
Repository record dc:identifier.uri
https://scholarwolf.unr.edu/handle/11714/11008

Chain of custody

source
Harvested from
University of Nevada - Reno
Base URL
scholarwolf.unr.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Paul, Shuvo Kumar. Enhancing Human-Robot Collaboration Through a Multi-Module Interaction Framework With Sensor Fusion: Object Recognition, Verbal Communication, User(s) of Interest Detection, Gesture and Gaze Recognition. Doctorate Degree thesis, University of Nevada, Reno, 2024. https://scholarwolf.unr.edu/handle/11714/11008