Back to search

Virginia Tech

AI-Driven Affective Captioning for Equitable STEM Access Among Deaf and Hard-of-Hearing Students

Abstract

dc:description.abstract

This dissertation investigates how Artificial Intelligebce (AI)- and Augmented Reality (AR)-supported captioning can improve communication access for Deaf and Hard-of-Hearing (DHH) learners in STEM contexts. Traditional real-time captions provide essential access to spoken language, but they often omit nonverbal and contextual information such as speaker identity, tone, emphasis, affect, and conversational intent. Across a preliminary design study and three empirical studies, this work examines how caption augmentations can preserve these missing layers of meaning while maintaining readability, timeliness, trust, and user control. The preliminary study compared traditional captions with emotion-augmented caption designs and showed that affective and visual cues can support comprehension when they are lightweight and text-centered, but may increase workload when they compete with the main transcript or visual scene. Study 1, a qualitative study with DHH participants, found that users valued emotion-aware captions when they clarified tone, emphasis, or speaker intent, but only when cues were timely, legible, optional, and subordinate to the transcript. Study 2 evaluated culturally adaptive emotive captioning in AR by comparing two cue formats: compact symbolic cues, implemented as emoji/icon indicators, and explicit textual affect labels, implemented as inline text-tags, across high- and low-context cultural cohorts. Compact symbolic cues produced a robust cross-cultural preference, while qualitative findings showed that participants valued the cues differently: some emphasized speed and reduced distraction, while others emphasized easier access to speaker emotion. Study 3 evaluated Speaker-Aware Affective Captioning, a multi-speaker captioning interface that combined speaker-attributed captions, confidence-gated affect tags, and an on-demand AI Describe feature. The study showed that speaker attribution was the most consistently valued support, while AI Describe helped users recover from missed or unclear information. Affect tags showed promise, but their usefulness depended on timing, persistence, interpretability, and trust. Across these studies, findings show that accessible captioning should not simply add more expressive information. Instead, next-generation captioning systems should reduce users' inferential burden through layered support: preserving the transcript first, identifying speakers, supporting recovery from missed information, and adding affective interpretation only when it is accurate, low-burden, and user-controllable. This dissertation contributes empirical evidence and design guidelines for trustworthy, culturally sensitive, and readable affective captioning systems for inclusive STEM learning.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy
Level thesis:degree_level
doctoral
Discipline thesis:degree_discipline
Computer Science & Applications
Department dc:contributor.department
Computer Science and#38; Applications
Grantor dc:publisher
Virginia Tech
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Ubur, Sunday David
Chair dc:contributor.committeechair
  • Gracanin, Denis
Committee members dc:contributor.committeemember
  • Rho, Ha Rim
  • Branham, Stacy Marie
  • Farghally, Mohammed Fawzi Seddik
  • Over, Sarah Elizabeth

Subjects

dc:subject × 5

Rights

dc:rights
Statement dc:rights
  • In Copyright
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Other
vt_gsexam:45961

Chain of custody

source
Harvested from
Virginia Tech
Base URL
vtechworks.lib.vt.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Ubur, Sunday David. AI-Driven Affective Captioning for Equitable STEM Access Among Deaf and Hard-of-Hearing Students. doctoral thesis, Virginia Tech, 2026. https://hdl.handle.net/10919/143221