Back to search

Georgia Institute of Technology

Robust and Flexible Reward Modeling for LLM Alignment

Abstract

dc:description.abstract

As large language models grow increasingly more capable, ensuring their alignment with human values is of utmost importance. One of the most promising ways to align language models is by designing a reward function that measures alignment with human values, and training the language model to maximize this reward. In this thesis, we focus on two approaches reward design: reward design from external feedback signals and reward learning from human annotated datasets. In this first chapter we develop a reward design framework, HERON, that eases reward function design by exploiting hierarchical relationships between feedback signals. In the second chapter, we propose an algorithm to learn reward functions from datasets with corrupted human annotations. In the last chapter, we develop an adversarial attack approach that automatically discovers flaws in state-of-the-art reward functions, and then subsequently use these attacks to train more robust reward models. Altogether, these contributions advance the scalability and robustness of reward modeling.

Degree

thesis:*
Level thesis:degree_level
Doctoral
Department dc:contributor.department
Industrial and Systems Engineering
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Bukharin, Alexander
Advisor dc:contributor.advisor
  • Zhao, Tuo
Committee members dc:contributor.committeemember
  • Zhang, Chao
  • Dai, Bo
  • Na, Sen
  • Delalleau, Olivier

Subjects

dc:subject × 3

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/77834

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Bukharin, Alexander. Robust and Flexible Reward Modeling for LLM Alignment. Doctoral thesis, Georgia Institute of Technology, 2025. https://hdl.handle.net/1853/77834