Back to search

University of North Texas

Implications of Punctuation Mark Normalization on Text Retrieval

Abstract

dc:description

This research investigated issues related to normalizing punctuation marks from a text retrieval perspective. A punctuated-centric approach was undertaken by exploring changes in meanings, whitespaces, words retrievability, and other issues related to normalizing punctuation marks. To investigate punctuation normalization issues, various frequency counts of punctuation marks and punctuation patterns were conducted using the text drawn from the Gutenberg Project archive and the Usenet Newsgroup archive. A number of useful punctuation mark types that could aid in analyzing punctuation marks were discovered. This study identified two types of punctuation normalization procedures: (1) lexical independent (LI) punctuation normalization and (2) lexical oriented (LO) punctuation normalization. Using these two types of punctuation normalization procedures, this study discovered various effects of punctuation normalization in terms of different search query types. By analyzing the punctuation normalization problem in this manner, a wide range of issues were discovered such as: the need to define different types of searching, to disambiguate the role of punctuation marks, to normalize whitespaces, and indexing of punctuated terms. This study concluded that to achieve the most positive effect in a text retrieval environment, normalizing punctuation marks should be based on an extensive systematic analysis of punctuation marks and punctuation patterns and their related factors. The results of this study indicate that there were many challenges due to complexity of language. Further, this study recommends avoiding a simplistic approach to punctuation normalization.

Degree

thesis:*
Grantor dc:publisher
University of North Texas
Year dc:date
2013

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Kim, Eungi
Contributors dc:contributor
  • Moen, William E.
  • O'Connor, Brian Clark
  • Wasson, Christina

Subjects

dc:subject × 5

Rights

dc:rights
Statement dc:rights
  • Public
  • Kim, Eungi
  • Copyright
  • Copyright is held by the author, unless otherwise noted. All rights Reserved.
Language dc:language
English

Identifiers

dc:identifier.*
Identifier
https://digital.library.unt.edu/ark:/67531/metadc500160/
ark: ark:/67531/metadc500160

Chain of custody

source
Harvested from
University of North Texas
Base URL
digital.library.unt.edu/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Kim, Eungi. Implications of Punctuation Mark Normalization on Text Retrieval. University of North Texas, 2013. https://doi.org/10.12794/metadc500160