Back to search

University of North Texas

Audo-Sight: Enabling Ambient Interaction for Blind and Visually Impaired Individuals

Abstract

dc:description

Visually impaired people face significant challenges when attempting to interact with and understand complex environments, and traditional assistive technologies often struggle to quickly provide necessary contextual understanding and interactive intelligence. This paper presents Audo-Sight, a state-of-the-art assistive system that seamlessly integrates Multimodal Large Language Models (MLLMs) to provide expedient, context-aware interactions for Blind and Visually Impaired (BVI) individuals. Unlike traditional assistive devices, our system can respond adaptively to the user environment, providing multimodal input processing through the fusion of visual, auditory, and textual information. The system operates in two different modalities: personalized interaction through user identification and public access in common spaces like museums and shopping malls. In tailored environments, the system adjusts its output to conform to the preferences of individual users, thus enhancing accessibility through a user-aware form of interaction. In shared environments, Audo-Sight employs a shared architecture that adapts to its current user with no manual reconfiguration required. To facilitate appropriate interactions with the LLM, the public Audo-Sight solution includes an Age-Range Determiner and Safe Query Filter. These restrict access to potentially inappropriate content for adolescent users. Additionally, the system ensures that responses are respectful to BVI users through NeMo Guardrails. By utilizing multimodal reasoning, BVI-cognizant response editing, and safeguarding features, this work represents a major leap in AI-driven accessibility technology capable of increasing autonomy, safety, and interaction for people with visual impairments in home and social settings. Finally, we present the integration of Audo-Sight and SmartSight, which enables enhanced situational awareness for BVI individuals. This integration takes advantage of the real-time visual analysis of SmartSight, combined with the extensive reasoning and interactive capabilities of Audo-Sight, and goes beyond object identification to provide context-driven, voice-controlled assistance in dynamic environments.

Degree

thesis:*
Grantor dc:publisher
University of North Texas
Year dc:date
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Ainary, Bhanuja
Contributors dc:contributor
  • Song Fu
  • Sagnik Ray Choudhury
  • Mohsen Amini Salehi
  • Ajita Rattani

Subjects

dc:subject × 5

Rights

dc:rights
Statement dc:rights
  • Public
  • Ainary, Bhanuja
  • Copyright
  • Copyright is held by the author, unless otherwise noted. All rights Reserved.
Language dc:language
English

Identifiers

dc:identifier.*
Identifier
local-cont-no: submission_4344
https://digital.library.unt.edu/ark:/67531/metadc2443352/
ark: ark:/67531/metadc2443352

Chain of custody

source
Harvested from
University of North Texas
Base URL
digital.library.unt.edu/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Ainary, Bhanuja. Audo-Sight: Enabling Ambient Interaction for Blind and Visually Impaired Individuals. University of North Texas, 2025. https://doi.org/10.12794/metadc2443352