Athens University Economics and Business (AUEB)
Word sense disambiguation and text relatedness based on word thesauri
Abstract
dc:descriptionAs the immense amount of text data increases rapidly over the years, the need to improve the quality of algorithms in text related tasks is eminent. Traditional mod- els for representing documents, like the standard vector space model, often neglect the aspect of semantic relatedness between words, suffering from the restriction of exact keywords matching, in order to explore the similarity or relatedness between segments of text. In critical tasks, like text classification and retrieval, which have been studied over the past decades intensively, this assumption of exact keyword matching is often the reason for poor performance. This thesis comes to explore the potential of incorporating semantic relatedness between documents in several text related applications, like text classification, retrieval and paraphrasing recognition. Several aspects have been taken into account, like word sense disambiguation to extract semantic information from text and the use of a word thesaurus, namely WordNet, in an effort to exhaust as many possibilities as possible in the workflow from analyzing and preprocessing documents up to embedding successfully the semantic information in a machine readable manner in those tasks. The outcome of this thesis shows that semantic information can be used efficiently in the studied tasks and that it can improve performance, widening the possibilities of more efficient algorithms in text applications. More precisely, in this thesis we propose a new measure of semantic relatedness that takes into account both the lexical semantic relatedness between words and their significance in the text that they are found (e.g., their TF-IDF scores). Such a measure that combines both aspects may help in many tasks, such as text classification and retrieval. The measure between words is based on their implicit semantic links. The approach does not require any type of training, since it exploits a word thesaurus, WordNet, in order to devise implicit semantic links between words. Based on this approach, we introduce a new measure of semantic relatedness between texts, which capitalizes on the semantic relatedness between individual words, and extends it to measure the relatedness between sets of words. We gradually validate our method: we first evaluate the performance of the measure between individual words in four data sets and then proceed with evaluating the performance of our method in measuring text-to-text semantic relatedness in three tasks. Experimental evaluation shows that the proposed method outperforms every other lexicon-based method of word semantic relatedness in the selected tasks and the tested data sets, and competes well against corpus-based approaches that require training.
Degree
thesis:*- Grantor dc:publisher
- Athens University Economics and Business (AUEB)
- Year dc:date
- 2009
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Τσατσαρώνης, Γεώργιος
- Tsatsaronis, Georgios
Subjects
dc:subject × 18- Αποσαφήνιση εννοιών λέξεων
- Μέτρα σημασιολογικής ομοιότητας και εγγύτητας
- Γλωσσικοί θησαυροί
- Κατηγοριοποίηση κειμένων
- Ανάκτηση πληροφοριών από κείμενα
- Σημασιολογικά δίκτυα
- Εξόρυξη γνώσης
- Word sense disambiguation
- Measures of lexical semantic relatedness and similarity
- Word thesauri
- Text classification
- Text retrieval
- Semantic networks
- Text and web mining
- Φυσικές Επιστήμες
- Επιστήμη Ηλεκτρονικών Υπολογιστών και Πληροφορική
- Natural Sciences
- Computer and Information Sciences
Rights
- Language dc:language
- eng
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/17724