Back to search

Dublin City University

Generation of Semantically Consistent Text and Its Evaluation

Abstract

dc:description.abstract

Generating semantically consistent text remains a crucial challenge in Natural Language Processing (NLP), especially in input-controlled settings such as data-to-text generation. Despite advances in neural language generation, current models often produce outputs that contain semantic inconsistencies with the input, including omissions, hallucinations, and distortions. These issues stem from the black-box, non-deterministic nature of neural models, which limits their interpretability and controllability. This thesis investigates how semantic consistency, the property that all information conveyed by the output is entailed by the input, can be more reliably evaluated and ultimately improved in generated text. Focusing on data-to-text generation, the thesis proposes that advancing semantic controllability requires a combination of principled formalisation of semantic errors and robust evaluation methods. First, it synthesises prior work on error annotation to develop a task-agnostic consensus taxonomy for semantic errors, distinguishing omission, addition, and substitution errors. Second, it applies this taxonomy in human evaluation studies, including span-based semantic error annotation to better understand the nature and reliability of human judgements. Third, it assesses the capacity of automatic evaluation methods, including lexical, embedding-based, and prompt-based approaches, to reflect human ratings of semantic consistency. In this context, the thesis explores LLM-as-judge setups for scoring system outputs and uses LLMs to sanity-check repeated human evaluation experiments, investigating their generalisability and reliability. Together, these contributions provide new tools insights, approaches and results for the fine-grained assessment of semantic consistency and lay the groundwork for more interpretable, reliable, and scalable evaluation frameworks in neural language generation.

Degree

thesis:*
Name dc:type.qualificationname
phd
Level dc:type.qualificationlevel
doctoral
Grantor dc:publisher.institution
Dublin City University
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Huidrom, Rudali

Rights

Language dc:language
en

Chain of custody

source
Harvested from
Dublin City University
Base URL
doras.dcu.ie/cgi/oai2
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Huidrom, Rudali. Generation of Semantically Consistent Text and Its Evaluation. doctoral thesis, Dublin City University, 2025.