Back to search

Cornell University

Characterizing the Difficulty of Natural Language Datasets for Machine Learning

Abstract

dc:description.abstract

Machine learning models can now achieve high performance on many natural language classification tasks. But we currently don't know how well a contemporary large language model will perform on a new task without directly trying it out. What makes a task difficult for machine learning models? We focus on the role of data---both a model's training data and that of downstream tasks---to go beyond evaluation performance in characterizing the difficulty of natural language tasks. We intervene throughout the language modeling pipeline, examining the interaction between a task's dataset and a) pretrained representations, b) evaluation, and c) pretraining data. We use random labelings to contextualize the degree of alignment between a task's data and a task's labels under different text representations. We use classifiers that guess uniformly at random, independently across examples, to contextualize a language model's performance on the small datasets typically used to evaluate in-context learning capabilities. We also examine the extent of evidence for the hypothesis that a downstream dataset’s similarity to a model’s pretraining dataset determines the model’s performance. Finally, we turn to case studies across image-text grounding, literary history, and architectural history where we are specifically interested in a model's performance on a given challenging dataset. Understanding the interaction between data and model will make our models ever more reliable on datasets that we care about, ultimately meeting text datasets where they are.

Degree

thesis:*
Name thesis:degree_name
Ph. D., Computer Science
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Computer Science
Grantor
Cornell University
Year dc:date.issued
2024

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Yauney, Gregory
Chair dc:contributor.chair
  • Mimno, David
Committee members dc:contributor.committeemember
  • Benson, Austin
  • Chattopadhyay, Eshan

Rights

Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Bibid
16612027
Dc Identifier Other
ProQuest Submission ID: 14600
ProQuest Publication ID: 31489088

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Yauney, Gregory. Characterizing the Difficulty of Natural Language Datasets for Machine Learning. Doctor of Philosophy thesis, Cornell University, 2024. https://hdl.handle.net/1813/116633