Back to search

Università degli studi di Trento

Strategies for addressing performance concerns and bias in designing, running, and reporting crowdsourcing experiment

Abstract

dc:description

Crowdsourcing involves releasing tasks on the internet for people with diverse backgrounds and skills to solve. Its adoption has come a long way, from scaling up problem-solving to becoming an environment for running complex experiments. Designing tasks to obtain reliable results is not straightforward as it requires many design choices that grow with the complexity of crowdsourcing projects, often demanding multiple trial-and-error iterations to properly configure. These inherent characteristics of crowdsourcing, the complexity of the design space, and heterogeneity of the crowd, set quality control as a major concern, making it an integral part of task design. Despite all the progress and guidelines for developing effective tasks, crowdsourcing still is addressed as an ``art'' rather than an exact science, in part due to the challenges related to task design but also because crowdsourcing allows more complex use cases nowadays, where the support available has not yet caught up with this progress. This leaves researchers and practitioners at the forefront to often rely on intuitions instead of informed decisions. Running controlled experiments in crowdsourcing platforms is a prominent example. Despite their importance, experiments in these platforms are not yet first-class citizens, making researchers resort to building custom features to compensate for the lack of support, where pitfalls in this process may be detrimental to the experimental outcome. In this thesis, therefore, our goal is to attend to the need of moving crowdsourcing from art to science from two perspectives that interplay with each other: providing guidance on task design through experimentation, and supporting the experimentation process itself. First, we select classification problems as a use case, given their importance and pervasive nature, and aim to bring awareness, empirical evidence, and guidance to previously unexplored task design choices to address performance concerns. And second, we also aim to make crowdsourcing accessible to researchers and practitioners from all backgrounds, reducing the requirement of in-depth knowledge of known biases in crowdsourcing platforms, experimental methods, as well as programming skills to overcome the limitations of crowdsourcing providers while running experiments. We start by proposing task design strategies to address workers' performance, quality and time, in crowdsourced classification tasks. Then we distill the challenges associated with running controlled crowdsourcing experiments, propose coping strategies to address these challenges, and introduce solutions to help researchers report their crowdsourcing experiments, moving crowdsourcing forward to standardized reporting.

Degree

thesis:*
Grantor dc:publisher
Università degli studi di Trento
Year dc:date
2021

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Ramirez Medina, Jorge Daniel
Contributors dc:contributor
  • Cernuzzi, Luca
  • Casati, Fabio

Subjects

dc:subject × 1

Rights

dc:rights
Statement dc:rights
  • info:eu-repo/semantics/openAccess
  • license:Altra licenza (Other type of license)
Language dc:language
eng

Identifiers

dc:identifier.*

Chain of custody

source
Harvested from
Università degli Studi di Trento
Base URL
iris.unitn.it/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Ramirez Medina, Jorge Daniel. Strategies for addressing performance concerns and bias in designing, running, and reporting crowdsourcing experiment. Università degli studi di Trento, 2021. http://hdl.handle.net/11572/321908