Georgia Institute of Technology
Scalable and Structured Evaluation of Large Language Models
Abstract
dc:description.abstractAs large language models (LLMs) move into real-world, open-ended applications, evaluating them becomes both more important and more difficult. This dissertation develops evaluation methods that go beyond multiple-choice accuracy to handle fine-grained text quality, sensitive-information measurement, multi-turn interaction, and long-context tasks. The chapters are organized as a sequence of self-contained projects that progress from learned automatic metrics to agent-based evaluation. I first present LENS, a learnable evaluation metric for text simplification trained on human ratings, and SALSA, an edit-based annotation framework for fine-grained, edit-level evaluation. I then develop models that detect and abstract self-disclosure spans in social media posts to reduce privacy risk. Next, I introduce SimulatorArena, a benchmark that assesses how closely LLM user simulators match real user behavior and how well their assistant evaluations align with human judgments. Finally, I introduce GAVEL for evaluating long-context legal summarization: GAVEL-Ref, a comprehensive reference-based framework built on checklist and residual-fact evaluation, and GAVEL-Agent, an agent scaffold for reference-free checklist extraction directly from source documents. Finally, I look ahead to the next generation of evaluation, where user simulators model people over the long term and autonomous agents carry out the evaluation themselves.
Degree
thesis:*- Name thesis:degree_name
- Computer Science, PhD
- Department thesis:degree_department
- Interactive Computing
- Grantor
- Georgia Institute of Technology
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Dou, Yao
- Advisor dc:contributor.advisor
-
- Xu, Wei
- Committee member dc:contributor.committeemember
-
- Chau, Duen Horng (Polo)
Rights
- Language dc:language.iso
- English
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/82194