Back to search

Virginia Tech

Expert-Driven Evaluation Platforms for Scientific Information Extraction Using Large Language Models

Abstract

dc:description.abstract

Researchers are increasingly adopting Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) to extract scientific information in structured formats for training ML models. However, generative models are stochastic and prone to hallucinations. This necessitates expert-driven evaluation to verify factual accuracy and scientific reasoning. Traditional evaluation workflows rely on generic survey tools and spreadsheets. These tools impose cognitive load and may introduce psychological biases into the evaluation process. This thesis presents the design, architecture, and deployment of two visual analytics platforms to address these cognitive and operational challenges in LLM evaluation. Specifically, these platforms support the evaluation of extraction pipelines developed for computational virology. These underlying computational models are designed to parse academic literature to identify mutations enabling zoonotic host shifts and to extract quantitative viral inactivation metrics. The VILLA Evaluation Platform provides a structured environment for the qualitative assessment of LLM-generated reasoning regarding viral mutations. By implementing interface strategies such as blind evaluation, randomized presentation, and side-by-side component layouts, VILLA mitigates anchoring and central tendency biases. The Persist Extraction Interface serves as a diagnostic tool for evaluating the extraction of structured, quantitative data. The Persist interface aligns model outputs directly with human-verified ground-truth datasets to capture feedback across different models and pipeline iterations. These process-oriented interfaces translate raw probabilistic outputs into the actionable diagnostic data required by machine learning engineers. By standardizing collaborative workflows, this work provides a scalable foundation for assessing and optimizing research agents in specialized scientific domains.

Degree

thesis:*
Name thesis:degree_name
Master of Science
Level thesis:degree_level
masters
Discipline thesis:degree_discipline
Computer Science & Applications
Department dc:contributor.department
Computer Science and#38; Applications
Grantor dc:publisher
Virginia Tech
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Gatne, Vasundhara Sagar
Chair dc:contributor.committeechair
  • Murali, T. M.
Committee members dc:contributor.committeemember
  • Hamouda, Sally
  • Karpatne, Anuj

Subjects

dc:subject × 5

Rights

dc:rights
Statement dc:rights
  • In Copyright
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Other
vt_gsexam:47303

Chain of custody

source
Harvested from
Virginia Tech
Base URL
vtechworks.lib.vt.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Gatne, Vasundhara Sagar. Expert-Driven Evaluation Platforms for Scientific Information Extraction Using Large Language Models. masters thesis, Virginia Tech, 2026. https://hdl.handle.net/10919/143654