Back to search

Cornell University

Practical Data Quality for Modern Data & Modern Uses, With Applications To America's COVID-19 Data

Abstract

dc:description.abstract

Modern data is often assumed to be of high quality. We dismiss this assumption using examples from America’s COVID-19 data. We explore the types and origins of these issues by diving into the data production process, noting that the root causes are not particular to this dataset but are typical of modern data in general. Data quality issues are frequently surprising, sometimes baffling, and often overwritten. We recover these issues by creating data releases which enable us to replay history. Using novel visualizations, we are able to surface quality issues within and across releases. Two such issues are of particular concern: major restatements, which rewrite history, and non-retroactive changes, which restart history. Data quality is defined by use case. We explore this through two applications: allocation of finite resources and surge prediction. While doing so, we propose κ-accuracy, a practical solution to a common obstacle, and argue for quality-based data selection. Research into data quality is urgently needed. The field is young, the theory uncoordinated, and the metrics all but nonexistent. Yet, assessing data quality is essential to using data properly. Researchers have developed so many ways to use data, and so few ways to assess whether or not we should. We hope this work will inspire research and investment into data quality.

Degree

thesis:*
Name thesis:degree_name
Ph. D., Statistics
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Statistics
Grantor
Cornell University
Year dc:date.issued
2023

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Frailey, Kerstin
Chair dc:contributor.chair
  • Wells, Martin
Committee members dc:contributor.committeemember
  • Joachims, Thorsten
  • Basu, Sumanta

Subjects

dc:subject × 6

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Bibid
16454723
Dc Identifier Other
ProQuest Submission ID: 14065
ProQuest Publication ID: 30811072

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Frailey, Kerstin. Practical Data Quality for Modern Data & Modern Uses, With Applications To America's COVID-19 Data. Doctor of Philosophy thesis, Cornell University, 2023. https://hdl.handle.net/1813/115679