Institutional Repository of Vilnius University
Debesų kompiuterijos sprendimais paremtas modernių saityno programų peržiūros robotas /
Abstract
dc:descriptionIn this paper, Web Crawling systems are investigated. Such systems are mostly used in Web archiving, gathering massive amounts of input data used in Machine Learning, also in search engi- nes – discovery of changing Web content is performed and detected resources are indexed. Desc- ribed systems perform automatic recursive crawling of Web resources while traveling through the graph of interconnected URL addresses. This paper tries to evaluate the possibility to implement such crawling systems that would be performance-useful and cost-effective, while fully relying on cloud computing infrastructure and its services. Also, the issue of crawling of modern, client-side rendered Web Applications is covered. The process of this paper work includes investigation of the domain of Web Crawling Sys- tems: boundaries of such systems are identified, architectual comparison of several widely known, publicly described Web Crawlers is made, different rendering models of websites and their impa- ct on Web Crawling process are presented. Practical paper work involves the implementation of cloud-based web crawler prototype and its cralwing capabilities evaluation during detailed cloud experiment. It was determined, that Broad Web Crawling, which focuses on massive content coverage is not possible with such cloud based implementations, however, small-to-medium focused crawls, which features rich content discovery, might be performed effectively with implemented architec- ture. Even though during the process of this paper work it was ascertained that any universal, bullet-proof way to identify website, which uses client-side rendering, is currently unknown, the effective probability model was presented, which allows to make such identification and perform selective, Javascript engine-based crawling – significant amount of new, previously undiscovered URL addresses is gathered.
Degree
thesis:*- Grantor dc:publisher
- Institutional Repository of Vilnius University
- Year dc:date
- 2020
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Taminskas, Kasparas,
- Contributors dc:contributor
-
- Šimkus, Aurimas
Rights
dc:rights- Statement dc:rights
-
- info:eu-repo/semantics/openAccess
- Language dc:language
- lit
Identifiers
dc:identifier.*- Repository record dc:identifier
- https://repository.vu.lt/VU:ELABAETD81704796&prefLang=en_US