Back to search

Institutional Repository of Vilnius University

Automatinis duomenų surinkimas naršyklėmis be grafinės sąsajos /

Abstract

dc:description

Web scraping has become an important part in the world of data-driven solutions. However, due to the continuous improvement of web technologies, web scraping becoming more and more difficult: modern pages load data dynamically or use web scraping prevention tools. One possible solution to this problem are headless browsers. The use of headless browsers makes it possible to simulate the behaviour of real users and to facilitate the collection of data from web pages that use prevention tools. However, web scrapers using headless browsers have a major drawback - they are costly to use on a large scale because of the large amount of resources required. For this reason, most of these types of systems are offered as paid services. In this paper, a web scraping system based on headless browsers is being developed. The system developed focuses on ease of use and on reducing the number of blocking resulting from browser fingerprinting. In this work, the web scraping process is reviewed, existing tools for the management of headless browsers analysed, and a web scraping is designed. The designed system is then implemented using the headless browser management library called Playwright. The evaluation of the designed system showed a reduction in the number of received blocks and CAPTCHA checks.

Degree

thesis:*
Grantor dc:publisher
Institutional Repository of Vilnius University
Year dc:date
2023

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Ramanauskas, Dominykas,

Rights

dc:rights
Statement dc:rights
  • info:eu-repo/semantics/openAccess
Language dc:language
lit

Identifiers

dc:identifier.*

Chain of custody

source
Harvested from
Vilnius University
Base URL
epublications.vu.lt/oai
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Ramanauskas, Dominykas,. Automatinis duomenų surinkimas naršyklėmis be grafinės sąsajos /. Institutional Repository of Vilnius University, 2023. https://repository.vu.lt/VU:ELABAETD192847232&prefLang=en_US