Cornell University
Practical Data Quality for Modern Data & Modern Uses, With Applications To America's COVID-19 Data
Abstract
dc:description.abstractModern data is often assumed to be of high quality. We dismiss this assumption using examples from America’s COVID-19 data. We explore the types and origins of these issues by diving into the data production process, noting that the root causes are not particular to this dataset but are typical of modern data in general. Data quality issues are frequently surprising, sometimes baffling, and often overwritten. We recover these issues by creating data releases which enable us to replay history. Using novel visualizations, we are able to surface quality issues within and across releases. Two such issues are of particular concern: major restatements, which rewrite history, and non-retroactive changes, which restart history. Data quality is defined by use case. We explore this through two applications: allocation of finite resources and surge prediction. While doing so, we propose κ-accuracy, a practical solution to a common obstacle, and argue for quality-based data selection. Research into data quality is urgently needed. The field is young, the theory uncoordinated, and the metrics all but nonexistent. Yet, assessing data quality is essential to using data properly. Researchers have developed so many ways to use data, and so few ways to assess whether or not we should. We hope this work will inspire research and investment into data quality.
Degree
thesis:*- Name thesis:degree_name
- Ph. D., Statistics
- Level thesis:degree_level
- Doctor of Philosophy
- Discipline thesis:degree_discipline
- Statistics
- Grantor
- Cornell University
- Year dc:date.issued
- 2023
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Frailey, Kerstin
- Chair dc:contributor.chair
-
- Wells, Martin
- Committee members dc:contributor.committeemember
-
- Joachims, Thorsten
- Basu, Sumanta
Subjects
dc:subject × 6Rights
dc:rights- Statement dc:rights
-
- Attribution 4.0 International
- Licence dc:rights.uri
- Language dc:language.iso
- en
Identifiers
dc:identifier.*- Dc Identifier Bibid
- 16454723
- Dc Identifier Other
-
ProQuest Submission ID: 14065
ProQuest Publication ID: 30811072