Back to search

Institutional Repository of Vilnius University

Debesų kompiuterijos sprendimais paremtas modernių saityno programų peržiūros robotas /

Abstract

dc:description

In this paper, Web Crawling systems are investigated. Such systems are mostly used in Web archiving, gathering massive amounts of input data used in Machine Learning, also in search engi- nes – discovery of changing Web content is performed and detected resources are indexed. Desc- ribed systems perform automatic recursive crawling of Web resources while traveling through the graph of interconnected URL addresses. This paper tries to evaluate the possibility to implement such crawling systems that would be performance-useful and cost-effective, while fully relying on cloud computing infrastructure and its services. Also, the issue of crawling of modern, client-side rendered Web Applications is covered. The process of this paper work includes investigation of the domain of Web Crawling Sys- tems: boundaries of such systems are identified, architectual comparison of several widely known, publicly described Web Crawlers is made, different rendering models of websites and their impa- ct on Web Crawling process are presented. Practical paper work involves the implementation of cloud-based web crawler prototype and its cralwing capabilities evaluation during detailed cloud experiment. It was determined, that Broad Web Crawling, which focuses on massive content coverage is not possible with such cloud based implementations, however, small-to-medium focused crawls, which features rich content discovery, might be performed effectively with implemented architec- ture. Even though during the process of this paper work it was ascertained that any universal, bullet-proof way to identify website, which uses client-side rendering, is currently unknown, the effective probability model was presented, which allows to make such identification and perform selective, Javascript engine-based crawling – significant amount of new, previously undiscovered URL addresses is gathered.

Degree

thesis:*
Grantor dc:publisher
Institutional Repository of Vilnius University
Year dc:date
2020

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Taminskas, Kasparas,
Contributors dc:contributor
  • Šimkus, Aurimas

Rights

dc:rights
Statement dc:rights
  • info:eu-repo/semantics/openAccess
Language dc:language
lit

Identifiers

dc:identifier.*

Chain of custody

source
Harvested from
Vilnius University
Base URL
epublications.vu.lt/oai
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Taminskas, Kasparas,. Debesų kompiuterijos sprendimais paremtas modernių saityno programų peržiūros robotas /. Institutional Repository of Vilnius University, 2020. https://repository.vu.lt/VU:ELABAETD81704796&prefLang=en_US