Back to search

Rochester Institute of Technology

Multimodal Brain-Computer Interface for Real-Scene Object Selection and Shared Robotic Arm Control

Abstract

dc:description.abstract

Due to the limited bandwidth and reliability of non-invasive electroencephalography (EEG)-based brain–computer interface (BCI) systems, it is difficult to control a six-degree-of-freedom (6-DoF) robotic arm using such a system. This thesis proposes a multimodal BCI system for controlling a robotic arm that divides control into three stages: initiating the task through right-hand motor imagery (MI), selecting an object of interest from a real scene using steady-state visual evoked potentials (SSVEPs) and webcam-based eye tracking, and autonomously performing the pick-and-drop task using a WidowX 250 robotic arm. Two separate offline experiments, each involving ten participants, were performed to test the SSVEP and MI components of the system. In the SSVEP experiment, several decoding pipelines were compared, and the effect of fusing eye tracking with SSVEPs was evaluated using different EEG window lengths. The TRCA-Net-Lite model with a gaze-attention module achieved a maximum accuracy of 98.00% using a 6.0-second window and a maximum information transfer rate of 48.01 bits/min using a 1.0-second window. In the MI experiment, several classifiers were compared for classifying right-hand MI, left-hand MI, and No-MI. DR-EEGNet achieved a three-class accuracy of 77.21% using subject-dependent four-fold cross-validation and was selected for online action initiation because of its compact structure. Online testing with six participants performing five trials each resulted in a first-attempt object-selection accuracy of 28/30 (93.3%). After the two incorrect selections were rejected using the double-blink mechanism and the object-selection stage was repeated, all 30 retained pick-and-drop tasks were successfully completed. The average task completion time was 49.52 ± 2.44 s. The main contributions of this thesis are a learned gaze-attention method for fusing SSVEP and webcam-derived gaze information, position-based selection that distinguishes visually identical objects, an MI-based arming gate for intentional task initiation, and a double-blink mechanism for resetting incorrect selections. This approach eliminates the need for continuous EEG-based movement control while retaining the user’s voluntary control over the task performed by the robotic arm.

Degree

thesis:*
Name thesis:degree_name
Electrical Engineering (MS)
Level thesis:degree_level
Thesis
Discipline thesis:degree_discipline
Electrical Engineering

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Keshtkar, Mohammad
Contributors dc:contributor
  • Jinane Mounsef

Subjects

dc:subject × 6

Identifiers

dc:identifier.*
Repository record dc:identifier
https://repository.rit.edu/theses/12784

Chain of custody

source
Harvested from
Rochester Institute of Technology
Base URL
repository.rit.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Keshtkar, Mohammad. Multimodal Brain-Computer Interface for Real-Scene Object Selection and Shared Robotic Arm Control. Thesis thesis, https://repository.rit.edu/theses/12784