Back to search

University of British Columbia

Exploring response patterns to open-ended items as indicators of data quality

Abstract

dc:description

Data collection through online sources such as Amazon’s Mechanical Turk (MTurk) is valuable to social science research. However, online sampling is susceptible to obtaining high rates of low quality data that are provided by individuals who are careless, fraudulent, and/or bots. Failure to identify and remove these problematic respondents can distort research results and lead to inaccurate conclusions. Current quality control strategies have challenges and limitations, including respondents learning to circumvent detection. Open-ended items have been found to offer rich insights into data quality and thus are useful for data quality assessments. Yet, more research is needed to better understand the usefulness of different open-ended items for data quality assessment purposes. The present study was a secondary analysis of a dataset comprising 343 participants recruited from MTurk to investigate the prevalence of problematic characteristics in response to four distinct open-ended items. Findings indicate that 86% of participants provided one or more low quality responses. These low quality responses tended to have multiple problematic characteristics, including duplication with other participants, vagueness, irrelevance, linguistical errors, and even blatant plagiarism. Importantly, the rates of these problematic characteristics varied depending on the specific open-ended item. A strict exclusionary approach led to a 14% retention rate whereas two different balanced approaches that allowed for some error had retention rates between 40% and 44%. These findings underscore the need for rigorous, yet pragmatic, detection methods and standards for online data collection, which should incorporate open-ended items. Recommendations for the use of open-ended items, including data handling procedures are discussed.

Degree

thesis:*
Name thesis:degree_name
Master of Arts - MA
Level thesis:degree_level
master's
Discipline thesis:degree_discipline
Measurement, Evaluation and Research Methodology
Grantor dc:publisher
University of British Columbia
Year dc:date
2023

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Webster, Alexis Danielle
Contributors dc:contributor
  • Hubley, Anita M.

Rights

dc:rights
Statement dc:rights
  • Attribution-NonCommercial-NoDerivatives 4.0 International
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/85765

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Webster, Alexis Danielle. Exploring response patterns to open-ended items as indicators of data quality. master's thesis, University of British Columbia, 2023. http://hdl.handle.net/2429/85765