Back to search

University of North Florida. School of Computing

Large language models for multimodal user interaction in a virtual environment

Abstract

dc:description.abstract

Virtual Reality (VR) is increasingly popular, but many barriers exist for individuals with little experience in coding, 3D modeling, or creating their own virtual experiences. The current tools used for content creation are often viewed as complex or frustrating, and they exhibit a steep learning curve. This problem presents an opportunity to develop tools incorporating Natural User Interfaces that better support end users. One such possible tool to assist users is Large Language Models (LLM), which can, extract a user's intention through speech or text. We posit how using LLMs can better support novice and expert developers alike, and using a human-in-the-loop approach, we can foster a Human-AI cocreative process. Towards the realization of that goal, we created a multimodal tool in which users can use a virtual reality system that incorporates a large language model, as well as direct manipulation, menus, eye gaze, and speech, to facilitate a more natural VR authoring experience. We created a template in Unity3D that can be customized for various tasks, including the construction of a 3D environment and the creation of commands. In this thesis, we describe a summative research study with 22 participants to determine the usability and future of our tool. Our participants were tasked with authoring a predefined environment, and we got a system usability scale score (SUS) of 57\% which is between "Ok" and "Good" which was expected due to the flexibility and high range of freedom within the system. All users indicated some degree of ease of use when using the system. However, most users also highlighted on how the mechanisms were difficult, highlighting the learning curve of the tool itself. Our results indicate that our multimodal approach, combining a large language model with other 3D user interface modalities, can provide a more intuitive and accessible interface for users. Future research and development will focus on fine-tuning these interactions and expanding the capabilities to better support the user.

Degree

thesis:*
Name thesis:degree_name
Master of Science in Computer and Information Sciences (MS)
Discipline thesis:degree_discipline
Computing
Grantor
University of North Florida. School of Computing
Date dc:date.issued
Fall

Author and committee

dc:creator, dc:contributor.*
Authors dc:creator
  • Sayed, Ahmed A
  • Sayed, Ahmed Ali Abdelkhalek, 1997-
Advisors dc:contributor.advisor
  • Pfeil, Kevin
  • Umapathy, Karthikeyan
  • Pittman, Corey
Contributors dc:contributor
  • Dr. Kevin Pfeil
  • Dr. Karthikeyan Umapathy
  • Dr. Corey Pittman

Subjects

dc:subject × 13

Rights

dc:rights

Identifiers

dc:identifier.*
Repository record dc:identifier
https://digitalcommons.unf.edu/etd/1300

Chain of custody

source
Harvested from
University of North Florida
Base URL
digitalcommons.unf.edu/do/oai/
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Sayed, Ahmed A; Sayed, Ahmed Ali Abdelkhalek, 1997-. Large language models for multimodal user interaction in a virtual environment. University of North Florida. School of Computing, https://digitalcommons.unf.edu/etd/1300