Back to search

University of North Texas

Methods to Evaluate and Improve the Quality of Text Summarization

Abstract

dc:description

Automatic text summarization is a beneficial tool to improve reading efficiency and information accessibility for users. It quickly condenses a large amount of information into key points, helping users to grasp essential information more easily. In addition, it supports various applications such as search engines, recommendation systems, and question-answering systems. However, the quality of automatically generated summaries is sometimes not satisfied. They can be hard to read, omit important contents of the original documents, or even more seriously contain hallucinations. Such issues make the generated summaries less useful and can even mislead the users. Therefore, evaluating the quality of summarization is highly essential. Summarization evaluation is still challenging as there are no ideal evaluation methods. The automatic method is fast, inexpensive, and can handle large volumes of data without human intervention. However, some studies demonstrate that existing automatic evaluation metrics are not well-aligned with human evaluation. Besides, the effectiveness of automatic evaluation metrics has not been thoroughly investigated. Additionally, summarization evaluation is subjective and varies based on users' expectations. As a result, the quality of summarization varies on the target users, indicating that evaluation methods should be customizable and adaptable to meet individual needs. However, conventional automatic evaluation metrics cannot be tailored to individual preferences. While human evaluation can be customized, it is time-consuming and costly. In this thesis, I comprehensively reevaluate existing automatic methods for assessing text summarization, with a focus on abstractive summarization. I also develop new methods to effectively assess the quality of model-generated summaries using LLMs. The proposed evaluation methods are customizable to different quality attributes that users are interested in. Additionally, the developed methods are designed to work in the case when no reference summaries are available. Furthermore, I introduce a new framework to iteratively improve the quality of LLM-based summarization by leveraging the evaluation results. The proposed methods are evaluated on multiple datasets with various domains. Also, I conduct additional analysis to verify the strengths and limitations of the proposed methods.

Degree

thesis:*
Grantor dc:publisher
University of North Texas
Year dc:date
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Nguyen, Huyen Thi Ngoc
Contributors dc:contributor
  • Eduardo Blanco
  • Kewei Sha
  • Ding, Junhua
  • Vincent Ng

Subjects

dc:subject × 4

Rights

dc:rights
Statement dc:rights
  • Public
  • Nguyen, Huyen Thi Ngoc
  • Copyright
  • Copyright is held by the author, unless otherwise noted. All rights Reserved.
Language dc:language
English

Identifiers

dc:identifier.*
Identifier
local-cont-no: submission_4378
https://digital.library.unt.edu/ark:/67531/metadc2443390/
ark: ark:/67531/metadc2443390

Chain of custody

source
Harvested from
University of North Texas
Base URL
digital.library.unt.edu/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Nguyen, Huyen Thi Ngoc. Methods to Evaluate and Improve the Quality of Text Summarization. University of North Texas, 2025. https://doi.org/10.12794/metadc2443390