University of North Texas
Methods to Evaluate and Improve the Quality of Text Summarization
Abstract
dc:descriptionAutomatic text summarization is a beneficial tool to improve reading efficiency and information accessibility for users. It quickly condenses a large amount of information into key points, helping users to grasp essential information more easily. In addition, it supports various applications such as search engines, recommendation systems, and question-answering systems. However, the quality of automatically generated summaries is sometimes not satisfied. They can be hard to read, omit important contents of the original documents, or even more seriously contain hallucinations. Such issues make the generated summaries less useful and can even mislead the users. Therefore, evaluating the quality of summarization is highly essential. Summarization evaluation is still challenging as there are no ideal evaluation methods. The automatic method is fast, inexpensive, and can handle large volumes of data without human intervention. However, some studies demonstrate that existing automatic evaluation metrics are not well-aligned with human evaluation. Besides, the effectiveness of automatic evaluation metrics has not been thoroughly investigated. Additionally, summarization evaluation is subjective and varies based on users' expectations. As a result, the quality of summarization varies on the target users, indicating that evaluation methods should be customizable and adaptable to meet individual needs. However, conventional automatic evaluation metrics cannot be tailored to individual preferences. While human evaluation can be customized, it is time-consuming and costly. In this thesis, I comprehensively reevaluate existing automatic methods for assessing text summarization, with a focus on abstractive summarization. I also develop new methods to effectively assess the quality of model-generated summaries using LLMs. The proposed evaluation methods are customizable to different quality attributes that users are interested in. Additionally, the developed methods are designed to work in the case when no reference summaries are available. Furthermore, I introduce a new framework to iteratively improve the quality of LLM-based summarization by leveraging the evaluation results. The proposed methods are evaluated on multiple datasets with various domains. Also, I conduct additional analysis to verify the strengths and limitations of the proposed methods.
Degree
thesis:*- Grantor dc:publisher
- University of North Texas
- Year dc:date
- 2025
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Nguyen, Huyen Thi Ngoc
- Contributors dc:contributor
-
- Eduardo Blanco
- Kewei Sha
- Ding, Junhua
- Vincent Ng
Subjects
dc:subject × 4Rights
dc:rights- Statement dc:rights
-
- Public
- Nguyen, Huyen Thi Ngoc
- Copyright
- Copyright is held by the author, unless otherwise noted. All rights Reserved.
- Language dc:language
- English
Identifiers
dc:identifier.*- Identifier
-
local-cont-no: submission_4378
https://digital.library.unt.edu/ark:/67531/metadc2443390/
ark: ark:/67531/metadc2443390