Back to search

University of Waterloo

What Do You Mean? Using Large Language Models for Semantic Evaluation of NL2SQL Queries

Abstract

dc:description.abstract

While significant research focuses on improving natural language to SQL (NL2SQL) translation, the evaluation of generated queries remains an understudied problem. Current metrics: Exact Match (EM) and Execution Accuracy (EA), fail to address critical nuances. EM prioritizes syntactic fidelity over semantic equivalence, penalizing valid query variations, while EA’s reliance on execution results introduces false positives when incorrect SQL coincidentally returns correct output due to dataset characteristics. These limitations make existing frameworks inadequate for real-world deployment, where robustness and intent preservation are of key importance. The advent of large language models (LLMs) offers new opportunities to improve these evaluation methodologies. We propose a hybrid framework that integrates execution based validation with LLM-driven semantic analysis to address these gaps. Our pipeline validates candidate SQL queries through EA, and applies Qwen 2.5 Coder, a 1.5B-parameter model optimized for code generation, to perform semantic equivalence checks. This two-fold process eliminates false positives by detecting logical inconsistencies (e.g., incorrect JOIN conditions or aliases) that EA overlooks. Crucially, Qwen 2.5 operates without database connectivity or schema linking, enabling schema-agnostic evaluation through cross-attention between natural language questions and SQL. Experiments on a combined dataset from Spider and other sources demonstrate that combining EA with Qwen 2.5’s generative reasoning achieves 94% accuracy. The framework identifies 100% of EA’s false positives. Ablation studies reveal that encoder models like Microsoft’s codeBERT perform best on simple SELECT-WHERE queries (94% F1), whereas Qwen 2.5 maintains 90% consistency across all complexity levels. This work establishes a new paradigm for NL2SQL evaluation.

Degree

thesis:*
Grantor dc:publisher
University of Waterloo
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Noor, Harrum

Subjects

dc:subject × 3

Rights

Language dc:language.iso
en

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/10012/21632

Chain of custody

source
Harvested from
University of Waterloo
Base URL
dspacemainprd01.lib.uwaterloo.ca/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Noor, Harrum. What Do You Mean? Using Large Language Models for Semantic Evaluation of NL2SQL Queries. University of Waterloo, 2025. https://hdl.handle.net/10012/21632