University of North Florida. School of Computing
Large language models for multimodal user interaction in a virtual environment
Abstract
dc:description.abstractVirtual Reality (VR) is increasingly popular, but many barriers exist for individuals with little experience in coding, 3D modeling, or creating their own virtual experiences. The current tools used for content creation are often viewed as complex or frustrating, and they exhibit a steep learning curve. This problem presents an opportunity to develop tools incorporating Natural User Interfaces that better support end users. One such possible tool to assist users is Large Language Models (LLM), which can, extract a user's intention through speech or text. We posit how using LLMs can better support novice and expert developers alike, and using a human-in-the-loop approach, we can foster a Human-AI cocreative process. Towards the realization of that goal, we created a multimodal tool in which users can use a virtual reality system that incorporates a large language model, as well as direct manipulation, menus, eye gaze, and speech, to facilitate a more natural VR authoring experience. We created a template in Unity3D that can be customized for various tasks, including the construction of a 3D environment and the creation of commands. In this thesis, we describe a summative research study with 22 participants to determine the usability and future of our tool. Our participants were tasked with authoring a predefined environment, and we got a system usability scale score (SUS) of 57\% which is between "Ok" and "Good" which was expected due to the flexibility and high range of freedom within the system. All users indicated some degree of ease of use when using the system. However, most users also highlighted on how the mechanisms were difficult, highlighting the learning curve of the tool itself. Our results indicate that our multimodal approach, combining a large language model with other 3D user interface modalities, can provide a more intuitive and accessible interface for users. Future research and development will focus on fine-tuning these interactions and expanding the capabilities to better support the user.
Degree
thesis:*- Name thesis:degree_name
- Master of Science in Computer and Information Sciences (MS)
- Discipline thesis:degree_discipline
- Computing
- Grantor
- University of North Florida. School of Computing
- Date dc:date.issued
- Fall
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Sayed, Ahmed A
- Sayed, Ahmed Ali Abdelkhalek, 1997-
- Advisors dc:contributor.advisor
-
- Pfeil, Kevin
- Umapathy, Karthikeyan
- Pittman, Corey
- Contributors dc:contributor
-
- Dr. Kevin Pfeil
- Dr. Karthikeyan Umapathy
- Dr. Corey Pittman
Subjects
dc:subject × 13- Thesis
- University of North Florida
- UNF
- Dissertations, Academic -- UNF -- Master of Science in Computer and Information Sciences
- Dissertations, Academic -- UNF -- Computing
- Large Language Models
- Virtual Reality
- Human Computer Interaction
- ChatGPT
- 3D Environment Creation
- Computer Engineering
- Engineering
- Virtual reality -- Design -- Computer programs; Interactive multimedia -- Authoring programs -- Technological innovations; Natural language processing (Computer science) -- Industrial applications; Artificial intelligence -- Computer programs -- Industrial applications; Multimodal user interfaces (Computer systems) -- Industrial applications
Rights
dc:rightsIdentifiers
dc:identifier.*- Repository record dc:identifier
- https://digitalcommons.unf.edu/etd/1300