Cornell University
From Silence to Voice: Sensing, Recognizing and Restoring Silent Speech
Abstract
dc:description.abstractSpeech is arguably the most natural form of human communication, yet millions of people cannot speak aloud, whether by choice in privacy-sensitive situations or by necessity after losing their voice to disease. This dissertation develops wearable systems that capture a person's silent articulatory movements and convert them into practical communication: text for those who choose silence, and restored voice for those who have lost theirs. Six systems form a progressive arc across three dimensions: sensing, recognition, and restoration. The first dimension is sensing. Miniature speakers on a wearable device emit inaudible sound waves toward the user's face. When the user silently mouths words, the movements of their lips, jaw, and tongue reshape the returning echoes into distinctive patterns that encode the articulatory state. SpeeChin, a necklace-mounted infrared camera, first proved that wearable silent speech recognition is feasible with over 50 commands but exposed the structural limits of optical sensing: high power consumption, privacy risk, and sensitivity to lighting. EchoSpeech replaced the camera with near-ultrasonic acoustic sensing on smart glasses, achieving 4.5% word error rate on 31 commands at 73.3 mW, a 74x power reduction. HPSpeech generalized acoustic sensing to commodity headphones, demonstrating that the approach works on devices millions of people already own. The second dimension is recognition. SoniSpeech constructed the first large-scale, open-vocabulary dataset for minimally-obtrusive wearable silent speech: 34.1 hours, 18,000 utterances, 5,356 unique words. A baseline system achieved 26.3% word error rate on silent speech with no language model, on a scaling curve that shows no saturation. WhisperWeave extends acoustic-sensing recognition to laryngectomees by fusing articulatory echo profiles with pseudo-whisper audio on a single pair of eyeglasses, achieving 13.8% word error rate through bimodal complementarity. The third dimension is restoration. ReVoice shifted from text output to voice synthesis, introducing bone conduction probing of the vocal tract and an end-to-end generative adversarial network that synthesizes natural speech directly from silent articulation with 102.7 ms latency. Evaluated with 26 participants including 11 people who had lost their vocal cords to laryngeal cancer, ReVoice achieved 18.9% word error rate and 3.82 mean opinion score for naturalness, with no significant difference between clinical and healthy participants. These results substantially exceed the speech quality of the electrolarynx while providing a hands-free, socially acceptable form factor. Together, these six systems demonstrate that active acoustic sensing on everyday wearables can sense silent articulation with sufficient fidelity. Powered by advanced machine learning, we can recognize silent speech from closed vocabulary to open-vocabulary natural language, generalize to clinical populations through bimodal fusion, and restore expressive voice for people with no vocal cords.
Degree
thesis:*- Name thesis:degree_name
- Ph. D., Information Science
- Level thesis:degree_level
- Doctor of Philosophy
- Discipline thesis:degree_discipline
- Information Science
- Grantor
- Cornell University
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Zhang, Ruidong
- Chair dc:contributor.chair
-
- Zhang, Cheng
- Committee members dc:contributor.committeemember
-
- Guimbretiere, Francois
- Choudhury, Tanzeem
Rights
dc:rights- Statement dc:rights
-
- Attribution-NonCommercial-ShareAlike 4.0 International
- Licence dc:rights.uri
- Language dc:language.iso
- en
Identifiers
dc:identifier.*- Dc Identifier Other
-
Zhang_cornellgrad_0058F_15582
https://www.proquest.com/LegacyDocView/DISSNUM/32676660