Back to search

Georgia Institute of Technology

Scalable and Structured Evaluation of Large Language Models

Abstract

dc:description.abstract

As large language models (LLMs) move into real-world, open-ended applications, evaluating them becomes both more important and more difficult. This dissertation develops evaluation methods that go beyond multiple-choice accuracy to handle fine-grained text quality, sensitive-information measurement, multi-turn interaction, and long-context tasks. The chapters are organized as a sequence of self-contained projects that progress from learned automatic metrics to agent-based evaluation. I first present LENS, a learnable evaluation metric for text simplification trained on human ratings, and SALSA, an edit-based annotation framework for fine-grained, edit-level evaluation. I then develop models that detect and abstract self-disclosure spans in social media posts to reduce privacy risk. Next, I introduce SimulatorArena, a benchmark that assesses how closely LLM user simulators match real user behavior and how well their assistant evaluations align with human judgments. Finally, I introduce GAVEL for evaluating long-context legal summarization: GAVEL-Ref, a comprehensive reference-based framework built on checklist and residual-fact evaluation, and GAVEL-Agent, an agent scaffold for reference-free checklist extraction directly from source documents. Finally, I look ahead to the next generation of evaluation, where user simulators model people over the long term and autonomous agents carry out the evaluation themselves.

Degree

thesis:*
Name thesis:degree_name
Computer Science, PhD
Department thesis:degree_department
Interactive Computing
Grantor
Georgia Institute of Technology
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Dou, Yao
Advisor dc:contributor.advisor
  • Xu, Wei
Committee member dc:contributor.committeemember
  • Chau, Duen Horng (Polo)

Rights

Language dc:language.iso
English

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/82194

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Dou, Yao. Scalable and Structured Evaluation of Large Language Models. Georgia Institute of Technology, 2026. https://hdl.handle.net/1853/82194