Back to search

Virginia Tech

Holistic and Generalizable Evaluation of Generative Models

Abstract

dc:description.abstract

The rapid advancement of generative models, e.g., language models and multimodal models, has created a critical gap between their sophisticated capabilities and the evaluation methodologies used to assess them. While these models now excel at complex creative and interactive tasks, evaluation approaches remain anchored to task-specific metrics designed for constrained generation. This dissertation addresses the fundamental challenge of developing comprehensive, scalable, and generalizable evaluation methodologies that match the complexity of modern generative models. First, I introduce our efforts to advance the evaluation paradigms for assessing emerging generative capabilities. I develop instruction-tuned evaluators that generalize to unseen evaluation aspects without retraining, enabling adaptive assessment with new criteria. I then extend evaluation to the multimodal scenario, where I introduce a holistic framework for unified text-and-image generation and address the unique challenges of cross-modality assessment. Next, I extend the evaluation of generative models to scientific domains and propose the structured ideation space that decomposes scientific ideas into orthogonal conceptual dimensions, enabling fine-grained novelty assessments grounded in retrieved evidence rather than surface-level similarity. Beyond assessing generative capabilities, I present our work in investigating generative models' safety and how to deploy them responsibly. I develop a systematic framework for evaluating safety risks in goal-driven interactions, revealing critical misalignments between models' refusal behaviors and their deployment of unethical strategies. I also introduce a generalizable guardrail training methodology that enables AI guardrails to adapt to unseen policies and domains. Together, these contributions establish new evaluation paradigms that enable more principled development of generative models while ensuring alignment with human values and societal needs.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy
Level thesis:degree_level
doctoral
Discipline thesis:degree_discipline
Computer Science & Applications
Department dc:contributor.department
Computer Science and#38; Applications
Grantor dc:publisher
Virginia Tech
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Liu, Minqian
Chairs dc:contributor.committeechair
  • Zhou, Dawei
  • Huang, Lifu
Committee members dc:contributor.committeemember
  • Ramakrishnan, Narendran
  • Lu, Chang Tien
  • North, Christopher L.
  • Gehrmann, Sebastian

Subjects

dc:subject × 3

Rights

dc:rights
Statement dc:rights
  • Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Other
vt_gsexam:46345

Chain of custody

source
Harvested from
Virginia Tech
Base URL
vtechworks.lib.vt.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Liu, Minqian. Holistic and Generalizable Evaluation of Generative Models. doctoral thesis, Virginia Tech, 2026. https://hdl.handle.net/10919/143067