Back to search

Cal Poly

KGScore-Open: Leveraging Knowledge Graph Semantics For Open-QA Evaluation

Abstract

dc:description.abstract

Evaluating active Question Answering (QA) systems, as users ask questions outside of the original testing data, has proven to be difficult, due to the difficulty of gauging answer quality without ground truth responses. We propose KGScore-Open, a configurable system capable of scoring questions and answers in Open Domain Question Answering (Open-QA) without ground truth answers present by leveraging DBPedia, a Knowledge Graph (KG) derived from Wikipedia, to score question-answer pairs. The system maps entities from questions and answers to DBPedia nodes, constructs a Knowledge Graph based on these entities, and calculates a relatedness score. Our system is validated on multiple datasets, achieving up to 83% accuracy in differentiating relevant from irrelevant answers in the Natural Questions dataset, 55% accuracy in classifying correct versus incorrect answers (hallucinations) in the TruthfulQA and HaluEval datasets, and 54% accuracy on the QA-Eval task using the EVOUNA dataset. The contributions of this work include a novel scoring system for indicating both relevancy and answer confidence in Open-QA without the need for ground truth answers, demonstrated efficacy across various tasks, and an extendable framework applicable to different KGs for evaluating QA systems of other domains.

Degree

thesis:*
Name thesis:degree_name
MS in Computer Science
Discipline thesis:degree_discipline
Computer Science

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Hausman, Nicholas
Contributors dc:contributor
  • Lubomir Stanchev
  • Computer Science
  • College of Engineering

Subjects

dc:subject × 5

Identifiers

dc:identifier.*
Repository record dc:identifier
https://digitalcommons.calpoly.edu/theses/2885

Chain of custody

source
Harvested from
Cal Poly
Base URL
digitalcommons.calpoly.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Hausman, Nicholas. KGScore-Open: Leveraging Knowledge Graph Semantics For Open-QA Evaluation. https://digitalcommons.calpoly.edu/theses/2885