Back to search

Aristotle University Of Thessaloniki (AUTH)

Web crawling, analysis and archiving

Abstract

dc:description

The Web is increasingly important for all aspects of our society, culture and economy. Webarchiving is the process of gathering digital materials from the Web, ingesting it, ensuringthat these materials are preserved in an archive, and making the collected materials availablefor future use and research. Web archiving is a difficult problem due to organizational andtechnical reasons. We focus on the technical aspects of Web archiving.In this dissertation, we focus on improving the data acquisition aspect of the Web archiving process. We establish the notion of Website Archivability (WA) and we introduce theCredible Live Evaluation of Archive Readiness Plus (CLEAR+) method to measure WA forany website. We propose new algorithms to optimise Web crawling using near-duplicatedetection and webgraph cycle detection, resolving also the problem of web spider traps.Following, we suggest that different types of websites demand different Web archiving approaches. We focus on social media and more specifically on weblogs. We introduce weblogarchiving as a special type of Web archiving and present our findings and developments inthis area: a technical survey of the blogosphere, a scalable approach to harvest modern weblogs and an integrated approach to preserve weblogs using a digital repository system.

Degree

thesis:*
Grantor dc:publisher
Aristotle University Of Thessaloniki (AUTH)
Year dc:date
2015

Author and committee

dc:creator, dc:contributor.*
Authors dc:creator
  • Banos, Vangelis
  • Μπάνος, Ευάγγελος

Subjects

dc:subject × 10

Rights

dc:rights
Statement dc:rights
  • BY
Language dc:language
eng

Identifiers

dc:identifier.*
Identifier
10.12681/eadd/36536

Chain of custody

source
Harvested from
Greek National Archive of PhD Theses
Base URL
phdtheses.ekt.gr/eadd_oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Banos, Vangelis; Μπάνος, Ευάγγελος. Web crawling, analysis and archiving. Aristotle University Of Thessaloniki (AUTH), 2015. http://hdl.handle.net/10442/hedi/36536