Back to search

CSUniversity San Bernardino

AN EMPIRICAL STUDY ON THE EFFICACY OF LLM-POWERED CHATBOTS IN BASIC INFORMATION RETRIEVAL TASKS

Abstract

dc:description.abstract

The rise of conversational user interfaces (CUIs) powered by large language models (LLMs) is transforming human-computer interaction. This study evaluates the efficacy of LLM-powered chatbots, trained on website data, compared to browsing websites for finding information about organizations across diverse sectors. A within-subjects experiment with 165 participants was conducted, involving similar information retrieval (IR) tasks using both websites (GUIs) and chatbots (CUIs). The research questions are: (Q1) Which interface helps users find information faster: LLM chatbots or websites? (Q2) Which interface helps users find more accurate information: LLM chatbots or websites?. The findings are: (Q1) Participants found information significantly faster using LLM-chatbots, Q2. Participants found more accurate information using LLM-chatbots. The conclusions are: (Q1) LLM-chatbots are highly efficient, and (Q2). LLM-chatbots are highly reliable for information lookup tasks. These findings highlight the potential of LLM-powered CUIs to revolutionize user experience and advocate for integrating advanced AI capabilities in future interface design. Future research should investigate: 1. LLM-chatbot interaction speed over time to measure efficiency especially with more complex questions, 2. The precision of these models over larger knowledge bases and complex questions, 3. Improvements in chatbot’s usability and its impact on user experience and human computer interaction (HCI), and 4. Gauge user preference over prolonged interactions over more complex questions.

Degree

thesis:*
Name thesis:degree_name
Master of Science in Information Systems and Technology
Level thesis:degree_level
Project
Discipline thesis:degree_discipline
Information and Decision Sciences

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Faysal, Naja
Contributors dc:contributor
  • Shayo, Conrad

Subjects

dc:subject × 12

Identifiers

dc:identifier.*
Repository record dc:identifier
https://scholarworks.lib.csusb.edu/etd/1938

Chain of custody

source
Harvested from
CSUniversity San Bernardino
Base URL
scholarworks.lib.csusb.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Faysal, Naja. AN EMPIRICAL STUDY ON THE EFFICACY OF LLM-POWERED CHATBOTS IN BASIC INFORMATION RETRIEVAL TASKS. Project thesis, https://scholarworks.lib.csusb.edu/etd/1938