{"id":{"repo_id":"njit","oai_identifier":"oai:digitalcommons.njit.edu:theses-4573"},"canonical_url":"https://search.ndltd.org/etd/njit/oai:digitalcommons.njit.edu:theses-4573","repository":{"repo_id":"njit","name":"NJIT","base_url":"https://digitalcommons.njit.edu/do/oai/"},"display":{"title":"Adaptive multimodal smart home control on a raspberry pi 5 using hand gestures, voice cues, and user feedback","abstract":"<p>A real time multimodal smart home control system deployed on a Raspberry Pi 5 is presented. The system combines hand gestures, short voice cues, and proximity aware interaction to execute household commands such as light brightness control, fan speed adjustment, and stop or kill switch actions. Lightweight gesture and keyword spotting voice classifiers were trained offline and exported to TensorFlow Lite for efficient on device inference. For more natural spoken phrases, the system additionally integrates a locally deployed pretrained offline ASR component rather than a speech recognizer trained from scratch. Using a USB camera and microphone, the system operates fully on the edge and translates accepted commands into GPIO(General Purpose Input/Output) based actuator control.</p><p>To improve robustness under changing real-world conditions, the decision layer uses adaptive fusion that begins with balanced weighting and shifts trust toward the modality that appears more reliable at runtime. Reliability is estimated using confidence and quality measures, including an audio signal to noise ratio-based score for voice and an image sharpness measure derived from Laplacian variance for gesture input. A physical feedback button provides live supervision to adjust fusion bias over time. A hybrid voice pathway also combines keyword spotting for fast short commands with an offline ASR fallback for more natural spoken phrases.</p><p>Evaluations under multiple conditions showed that the keyword spotting based multimodal pathway remained more stable across environments, while offline ASR performed very well in quiet settings but degraded significantly under noise.</p>","abstract_html":"&lt;p&gt;A real time multimodal smart home control system deployed on a Raspberry Pi 5 is presented. The system combines hand gestures, short voice cues, and proximity aware interaction to execute household commands such as light brightness control, fan speed adjustment, and stop or kill switch actions. Lightweight gesture and keyword spotting voice classifiers were trained offline and exported to TensorFlow Lite for efficient on device inference. For more natural spoken phrases, the system additionally integrates a locally deployed pretrained offline ASR component rather than a speech recognizer trained from scratch. Using a USB camera and microphone, the system operates fully on the edge and translates accepted commands into GPIO(General Purpose Input/Output) based actuator control.&lt;/p&gt;&lt;p&gt;To improve robustness under changing real-world conditions, the decision layer uses adaptive fusion that begins with balanced weighting and shifts trust toward the modality that appears more reliable at runtime. Reliability is estimated using confidence and quality measures, including an audio signal to noise ratio-based score for voice and an image sharpness measure derived from Laplacian variance for gesture input. A physical feedback button provides live supervision to adjust fusion bias over time. A hybrid voice pathway also combines keyword spotting for fast short commands with an offline ASR fallback for more natural spoken phrases.&lt;/p&gt;&lt;p&gt;Evaluations under multiple conditions showed that the keyword spotting based multimodal pathway remained more stable across environments, while offline ASR performed very well in quiet settings but degraded significantly under noise.&lt;/p&gt;","abstract_has_math":false,"creators":["Bora, Vaibhav"],"institution":null,"degree_name":"Master of Science in Artificial Intelligence - (M.S.)","degree_level":null,"degree_discipline":"Data Science","degree_department":null,"school":null,"contributors":["Huiran Jin","Lijing Wang","Alisha Pradhan"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":null,"date_issued":"","date_published":null,"updated_at":"2026-11-19T08:44:11Z","subjects":["Multimodal smart home control","Gesture recognition","Keyword spotting (KWS)","Sensor fusion","Edge AI (Raspberry Pi 5)","User feedback","Artificial Intelligence and Robotics","Robotics"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[],"person_identifier_entries":[]},"links":{"outbound_url":"https://digitalcommons.njit.edu/theses/3549","outbound_label":"Repository record","outbound_source":"dc:identifier","record_url":"https://digitalcommons.njit.edu/theses/3549","record_label":"Repository record","record_kind":"record","record_source":"dc:identifier","download_url":null,"download_label":null,"download_source":null,"access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://digitalcommons.njit.edu/do/oai/?verb=GetRecord&metadataPrefix=oai_etdms&identifier=oai%3Adigitalcommons.njit.edu%3Atheses-4573","prefix":"oai_etdms"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Huiran Jin","Lijing Wang","Alisha Pradhan"]},{"key":"dc:creator","label":"Author","values":["Bora, Vaibhav"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.created","label":"Dc Date Created","values":["2026-05-31T07:00:00Z"]},{"key":"dc:type","label":"Dc Type","values":["Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Data Science"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Master of Science in Artificial Intelligence - (M.S.)"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Multimodal smart home control","Gesture recognition","Keyword spotting (KWS)","Sensor fusion","Edge AI (Raspberry Pi 5)","User feedback","Artificial Intelligence and Robotics","Robotics"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://digitalcommons.njit.edu/theses/3549"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["<p>A real time multimodal smart home control system deployed on a Raspberry Pi 5 is presented. The system combines hand gestures, short voice cues, and proximity aware interaction to execute household commands such as light brightness control, fan speed adjustment, and stop or kill switch actions. Lightweight gesture and keyword spotting voice classifiers were trained offline and exported to TensorFlow Lite for efficient on device inference. For more natural spoken phrases, the system additionally integrates a locally deployed pretrained offline ASR component rather than a speech recognizer trained from scratch. Using a USB camera and microphone, the system operates fully on the edge and translates accepted commands into GPIO(General Purpose Input/Output) based actuator control.</p><p>To improve robustness under changing real-world conditions, the decision layer uses adaptive fusion that begins with balanced weighting and shifts trust toward the modality that appears more reliable at runtime. Reliability is estimated using confidence and quality measures, including an audio signal to noise ratio-based score for voice and an image sharpness measure derived from Laplacian variance for gesture input. A physical feedback button provides live supervision to adjust fusion bias over time. A hybrid voice pathway also combines keyword spotting for fast short commands with an offline ASR fallback for more natural spoken phrases.</p><p>Evaluations under multiple conditions showed that the keyword spotting based multimodal pathway remained more stable across environments, while offline ASR performed very well in quiet settings but degraded significantly under noise.</p>"]},{"key":"dc:title","label":"Title","values":["Adaptive multimodal smart home control on a raspberry pi 5 using hand gestures, voice cues, and user feedback"]}]}],"canonical_facts":{"dc:contributor":["Huiran Jin","Lijing Wang","Alisha Pradhan"],"dc:creator":["Bora, Vaibhav"],"dc:date.created":["2026-05-31T07:00:00Z"],"dc:description.abstract":["<p>A real time multimodal smart home control system deployed on a Raspberry Pi 5 is presented. The system combines hand gestures, short voice cues, and proximity aware interaction to execute household commands such as light brightness control, fan speed adjustment, and stop or kill switch actions. Lightweight gesture and keyword spotting voice classifiers were trained offline and exported to TensorFlow Lite for efficient on device inference. For more natural spoken phrases, the system additionally integrates a locally deployed pretrained offline ASR component rather than a speech recognizer trained from scratch. Using a USB camera and microphone, the system operates fully on the edge and translates accepted commands into GPIO(General Purpose Input/Output) based actuator control.</p><p>To improve robustness under changing real-world conditions, the decision layer uses adaptive fusion that begins with balanced weighting and shifts trust toward the modality that appears more reliable at runtime. Reliability is estimated using confidence and quality measures, including an audio signal to noise ratio-based score for voice and an image sharpness measure derived from Laplacian variance for gesture input. A physical feedback button provides live supervision to adjust fusion bias over time. A hybrid voice pathway also combines keyword spotting for fast short commands with an offline ASR fallback for more natural spoken phrases.</p><p>Evaluations under multiple conditions showed that the keyword spotting based multimodal pathway remained more stable across environments, while offline ASR performed very well in quiet settings but degraded significantly under noise.</p>"],"dc:identifier":["https://digitalcommons.njit.edu/theses/3549"],"dc:subject":["Multimodal smart home control","Gesture recognition","Keyword spotting (KWS)","Sensor fusion","Edge AI (Raspberry Pi 5)","User feedback","Artificial Intelligence and Robotics","Robotics"],"dc:title":["Adaptive multimodal smart home control on a raspberry pi 5 using hand gestures, voice cues, and user feedback"],"dc:type":["Thesis"],"thesis:degree_discipline":["Data Science"],"thesis:degree_name":["Master of Science in Artificial Intelligence - (M.S.)"]},"updated_at":"2026-11-19T08:44:11Z"}