Back to search

University of Washington

Understanding and Addressing Uncertainty of the Crowd

Abstract

dc:description.abstract

Uncertainty is crucial to understanding human judgments. Whether it's annotators producing data used to train and evaluate machine learning systems, teaching staff assigning grades to open-ended student responses, or online communities adjudicating the moderation action to apply to a piece of content, many groups and individuals need to account for and address the uncertainty that comes along with making judgments. As the application of computing technology expands to more areas of society, groups and individuals are faced with the need to make judgments on increasingly complex, subjective, and nuanced tasks at scale. This dissertation presents a set of novel tools and processes that improve upon how we capture, distinguish, and address various sources of uncertainty present in individual and collective human judgments. I will start by introducing, Goldilocks, a tool for conducting scalar rating annotations that enables different sources of uncertainty to be distinguished while also improving consistency. Then I will introduce case law crowdsourcing as a process that enables capturing similar insights about uncertainty on complex categorical classification judgment tasks by utilizing prior decisions in the form of precedent cases. Following this, I will present Cicero, a tool that addresses one specific source of uncertainty -- disagreement -- through multi-turn, contextual deliberation. Finally, I will tie together individual tools for understanding and addressing uncertainty through a dynamic workflow that applies a targeted intervention on a per-instance scale to reduce uncertainty using measurements that allow us to distinguish the source of uncertainty. I conclude by discussing the limitations of current tools and give some insights for future work in designing new tools and processes that natively support judgment under uncertainty.

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Chen, Quan Ze
Advisor dc:contributor.advisor
  • Zhang, Amy X

Subjects

dc:subject × 6

Rights

dc:rights
Statement dc:rights
  • CC BY-SA
Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
http://hdl.handle.net/1773/49879

Chain of custody

source
Harvested from
University of Washington
Base URL
digital.lib.washington.edu/server/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Chen, Quan Ze. Understanding and Addressing Uncertainty of the Crowd. 2023. http://hdl.handle.net/1773/49879