Cal Poly
Designing for Trust in Chat-Based Question Answering Systems: An Exchange-Based Retrieval Approach
Abstract
dc:description.abstractCommunity chat platforms such as Discord and Slack support spontaneous, collaborative communication but make it difficult to retrieve previously discussed information. As conversations accumulate, valuable exchanges become buried, leading to repeated questions and sustained burden on experienced community members. This work contributes a set of design requirements for question-answering systems operating over unstructured chat data, a Discord bot prototype implementing those requirements named Echo, and an empirical evaluation of how such a system affects user trust. Rather than encoding discrete question-answer pairs or generating synthetic responses with a language model, Echo indexes conversation topics for semantic retrieval and presents results as constructed exchanges of original messages attributed to their authors. This design preserves conversational context and grounds responses in source material. A user study evaluated how interaction with Echo affected trust-related perceptions of automated question-answering systems. Ten participants performed information-seeking tasks within a synthetic Discord server and completed trust surveys before and after interaction. All participants exhibited increased composite trust scores following use of the system, with a mean increase of 8.4 points. Findings suggest high perceived value in semantic retrieval over keyword search and in exchange-based responses, as well as the importance of source attribution in building user trust.
Degree
thesis:*- Name thesis:degree_name
- MS in Computer Science
- Discipline thesis:degree_discipline
- Computer Science
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- McCutchen, Nathan
- Contributors dc:contributor
-
- Ayaan Kazerouni
- Computer Science
- College of Engineering
Subjects
dc:subject × 8Identifiers
dc:identifier.*- Repository record dc:identifier
- https://digitalcommons.calpoly.edu/theses/3245