CSUniversity San Bernardino
AN EMPIRICAL STUDY ON THE EFFICACY OF LLM-POWERED CHATBOTS IN BASIC INFORMATION RETRIEVAL TASKS
Abstract
dc:description.abstractThe rise of conversational user interfaces (CUIs) powered by large language models (LLMs) is transforming human-computer interaction. This study evaluates the efficacy of LLM-powered chatbots, trained on website data, compared to browsing websites for finding information about organizations across diverse sectors. A within-subjects experiment with 165 participants was conducted, involving similar information retrieval (IR) tasks using both websites (GUIs) and chatbots (CUIs). The research questions are: (Q1) Which interface helps users find information faster: LLM chatbots or websites? (Q2) Which interface helps users find more accurate information: LLM chatbots or websites?. The findings are: (Q1) Participants found information significantly faster using LLM-chatbots, Q2. Participants found more accurate information using LLM-chatbots. The conclusions are: (Q1) LLM-chatbots are highly efficient, and (Q2). LLM-chatbots are highly reliable for information lookup tasks. These findings highlight the potential of LLM-powered CUIs to revolutionize user experience and advocate for integrating advanced AI capabilities in future interface design. Future research should investigate: 1. LLM-chatbot interaction speed over time to measure efficiency especially with more complex questions, 2. The precision of these models over larger knowledge bases and complex questions, 3. Improvements in chatbot’s usability and its impact on user experience and human computer interaction (HCI), and 4. Gauge user preference over prolonged interactions over more complex questions.
Degree
thesis:*- Name thesis:degree_name
- Master of Science in Information Systems and Technology
- Level thesis:degree_level
- Project
- Discipline thesis:degree_discipline
- Information and Decision Sciences
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Faysal, Naja
- Contributors dc:contributor
-
- Shayo, Conrad
Subjects
dc:subject × 12- Large Language Models (LLMs)
- Conversational User Interfaces (CUIs)
- Information Retrieval (IR)
- User Experience (UX)
- Graphical User Interfaces (GUIs)
- Human-Computer Interaction (HCI)
- Artificial Intelligence and Robotics
- Databases and Information Systems
- Graphic Design
- Graphics and Human Computer Interfaces
- Industrial and Product Design
- Interactive Arts
Identifiers
dc:identifier.*- Repository record dc:identifier
- https://scholarworks.lib.csusb.edu/etd/1938