Back to search

Cornell University

Improving Data Efficiency and Availability for Alignment

Abstract

dc:description.abstract

AI systems have become increasingly powerful by using increasing amounts of compute in order to implement more and more complex tasks. For example, writing a rules-based search engine that explicitly computes what links a specific user would be interested in for a given query is prohibitively difficult, but machine learning can predict what links a user is most likely to click on. Reinforcement learning from human feedback takes ML further by learning a reward model from preference comparisons to improve an LLM's responses, rather than trying to specify good behavior directly. In both of these settings, human feedback is critical to aligning ML systems for personalization and to perform difficult-to-directly-specify tasks. While compute gets cheaper and cheaper, human attention stays expensive. How can human feedback be more efficient? How can it be more available? This thesis presents several projects which improve data efficiency and data availability for aligning AI systems using human feedback. First, it analyzes the exploration and exploitation tradeoff induced by introducing an explicit cost for observing rewards in a bandit setting. Second, it provides methods for allocating a fixed interaction budget to improve the data efficiency of off-policy learning and evaluation in realistic settings appropriate for search and recommendation systems. Third, it shows how to improve LLM-based assistants using implicit feedback gathered from user interactions. Finally, it outlines preliminary results and promising directions for future work for using human feedback.

Degree

thesis:*
Name thesis:degree_name
Ph. D., Computer Science
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Computer Science
Grantor
Cornell University
Year dc:date.issued
2024

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Tucker, Aaron
Chair dc:contributor.chair
  • Joachims, Thorsten
Committee members dc:contributor.committeemember
  • Sun, Wen
  • Weinberger, Kilian

Subjects

dc:subject × 5

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Bibid
16922020
Dc Identifier Other
ProQuest Submission ID: 14777
ProQuest Publication ID: 31641406

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Tucker, Aaron. Improving Data Efficiency and Availability for Alignment. Doctor of Philosophy thesis, Cornell University, 2024. https://hdl.handle.net/1813/117238