Back to search

Cal Poly

Designing for Trust in Chat-Based Question Answering Systems: An Exchange-Based Retrieval Approach

Abstract

dc:description.abstract

Community chat platforms such as Discord and Slack support spontaneous, collaborative communication but make it difficult to retrieve previously discussed information. As conversations accumulate, valuable exchanges become buried, leading to repeated questions and sustained burden on experienced community members. This work contributes a set of design requirements for question-answering systems operating over unstructured chat data, a Discord bot prototype implementing those requirements named Echo, and an empirical evaluation of how such a system affects user trust. Rather than encoding discrete question-answer pairs or generating synthetic responses with a language model, Echo indexes conversation topics for semantic retrieval and presents results as constructed exchanges of original messages attributed to their authors. This design preserves conversational context and grounds responses in source material. A user study evaluated how interaction with Echo affected trust-related perceptions of automated question-answering systems. Ten participants performed information-seeking tasks within a synthetic Discord server and completed trust surveys before and after interaction. All participants exhibited increased composite trust scores following use of the system, with a mean increase of 8.4 points. Findings suggest high perceived value in semantic retrieval over keyword search and in exchange-based responses, as well as the importance of source attribution in building user trust.

Degree

thesis:*
Name thesis:degree_name
MS in Computer Science
Discipline thesis:degree_discipline
Computer Science

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • McCutchen, Nathan
Contributors dc:contributor
  • Ayaan Kazerouni
  • Computer Science
  • College of Engineering

Subjects

dc:subject × 8

Identifiers

dc:identifier.*
Repository record dc:identifier
https://digitalcommons.calpoly.edu/theses/3245

Chain of custody

source
Harvested from
Cal Poly
Base URL
digitalcommons.calpoly.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

McCutchen, Nathan. Designing for Trust in Chat-Based Question Answering Systems: An Exchange-Based Retrieval Approach. https://digitalcommons.calpoly.edu/theses/3245