Back to search

NJIT

Adaptive multimodal smart home control on a raspberry pi 5 using hand gestures, voice cues, and user feedback

Abstract

dc:description.abstract

<p>A real time multimodal smart home control system deployed on a Raspberry Pi 5 is presented. The system combines hand gestures, short voice cues, and proximity aware interaction to execute household commands such as light brightness control, fan speed adjustment, and stop or kill switch actions. Lightweight gesture and keyword spotting voice classifiers were trained offline and exported to TensorFlow Lite for efficient on device inference. For more natural spoken phrases, the system additionally integrates a locally deployed pretrained offline ASR component rather than a speech recognizer trained from scratch. Using a USB camera and microphone, the system operates fully on the edge and translates accepted commands into GPIO(General Purpose Input/Output) based actuator control.</p><p>To improve robustness under changing real-world conditions, the decision layer uses adaptive fusion that begins with balanced weighting and shifts trust toward the modality that appears more reliable at runtime. Reliability is estimated using confidence and quality measures, including an audio signal to noise ratio-based score for voice and an image sharpness measure derived from Laplacian variance for gesture input. A physical feedback button provides live supervision to adjust fusion bias over time. A hybrid voice pathway also combines keyword spotting for fast short commands with an offline ASR fallback for more natural spoken phrases.</p><p>Evaluations under multiple conditions showed that the keyword spotting based multimodal pathway remained more stable across environments, while offline ASR performed very well in quiet settings but degraded significantly under noise.</p>

Degree

thesis:*
Name thesis:degree_name
Master of Science in Artificial Intelligence - (M.S.)
Discipline thesis:degree_discipline
Data Science

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Bora, Vaibhav
Contributors dc:contributor
  • Huiran Jin
  • Lijing Wang
  • Alisha Pradhan

Subjects

dc:subject × 8

Identifiers

dc:identifier.*
Repository record dc:identifier
https://digitalcommons.njit.edu/theses/3549

Chain of custody

source
Harvested from
NJIT
Base URL
digitalcommons.njit.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Bora, Vaibhav. Adaptive multimodal smart home control on a raspberry pi 5 using hand gestures, voice cues, and user feedback. https://digitalcommons.njit.edu/theses/3549