Georgia Institute of Technology
Robust and Flexible Reward Modeling for LLM Alignment
Abstract
dc:description.abstractAs large language models grow increasingly more capable, ensuring their alignment with human values is of utmost importance. One of the most promising ways to align language models is by designing a reward function that measures alignment with human values, and training the language model to maximize this reward. In this thesis, we focus on two approaches reward design: reward design from external feedback signals and reward learning from human annotated datasets. In this first chapter we develop a reward design framework, HERON, that eases reward function design by exploiting hierarchical relationships between feedback signals. In the second chapter, we propose an algorithm to learn reward functions from datasets with corrupted human annotations. In the last chapter, we develop an adversarial attack approach that automatically discovers flaws in state-of-the-art reward functions, and then subsequently use these attacks to train more robust reward models. Altogether, these contributions advance the scalability and robustness of reward modeling.
Degree
thesis:*- Level thesis:degree_level
- Doctoral
- Department dc:contributor.department
- Industrial and Systems Engineering
- Grantor dc:publisher
- Georgia Institute of Technology
- Year dc:date.issued
- 2025
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Bukharin, Alexander
- Advisor dc:contributor.advisor
-
- Zhao, Tuo
- Committee members dc:contributor.committeemember
-
- Zhang, Chao
- Dai, Bo
- Na, Sen
- Delalleau, Olivier
Subjects
dc:subject × 3Rights
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/77834