NJIT
Adaptive multimodal smart home control on a raspberry pi 5 using hand gestures, voice cues, and user feedback
Abstract
dc:description.abstract<p>A real time multimodal smart home control system deployed on a Raspberry Pi 5 is presented. The system combines hand gestures, short voice cues, and proximity aware interaction to execute household commands such as light brightness control, fan speed adjustment, and stop or kill switch actions. Lightweight gesture and keyword spotting voice classifiers were trained offline and exported to TensorFlow Lite for efficient on device inference. For more natural spoken phrases, the system additionally integrates a locally deployed pretrained offline ASR component rather than a speech recognizer trained from scratch. Using a USB camera and microphone, the system operates fully on the edge and translates accepted commands into GPIO(General Purpose Input/Output) based actuator control.</p><p>To improve robustness under changing real-world conditions, the decision layer uses adaptive fusion that begins with balanced weighting and shifts trust toward the modality that appears more reliable at runtime. Reliability is estimated using confidence and quality measures, including an audio signal to noise ratio-based score for voice and an image sharpness measure derived from Laplacian variance for gesture input. A physical feedback button provides live supervision to adjust fusion bias over time. A hybrid voice pathway also combines keyword spotting for fast short commands with an offline ASR fallback for more natural spoken phrases.</p><p>Evaluations under multiple conditions showed that the keyword spotting based multimodal pathway remained more stable across environments, while offline ASR performed very well in quiet settings but degraded significantly under noise.</p>
Degree
thesis:*- Name thesis:degree_name
- Master of Science in Artificial Intelligence - (M.S.)
- Discipline thesis:degree_discipline
- Data Science
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Bora, Vaibhav
- Contributors dc:contributor
-
- Huiran Jin
- Lijing Wang
- Alisha Pradhan
Subjects
dc:subject × 8Identifiers
dc:identifier.*- Repository record dc:identifier
- https://digitalcommons.njit.edu/theses/3549