Institutional Repository of Vilnius University
Automatinis duomenų surinkimas naršyklėmis be grafinės sąsajos /
Abstract
dc:descriptionWeb scraping has become an important part in the world of data-driven solutions. However, due to the continuous improvement of web technologies, web scraping becoming more and more difficult: modern pages load data dynamically or use web scraping prevention tools. One possible solution to this problem are headless browsers. The use of headless browsers makes it possible to simulate the behaviour of real users and to facilitate the collection of data from web pages that use prevention tools. However, web scrapers using headless browsers have a major drawback - they are costly to use on a large scale because of the large amount of resources required. For this reason, most of these types of systems are offered as paid services. In this paper, a web scraping system based on headless browsers is being developed. The system developed focuses on ease of use and on reducing the number of blocking resulting from browser fingerprinting. In this work, the web scraping process is reviewed, existing tools for the management of headless browsers analysed, and a web scraping is designed. The designed system is then implemented using the headless browser management library called Playwright. The evaluation of the designed system showed a reduction in the number of received blocks and CAPTCHA checks.
Degree
thesis:*- Grantor dc:publisher
- Institutional Repository of Vilnius University
- Year dc:date
- 2023
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Ramanauskas, Dominykas,
Rights
dc:rights- Statement dc:rights
-
- info:eu-repo/semantics/openAccess
- Language dc:language
- lit
Identifiers
dc:identifier.*- Repository record dc:identifier
- https://repository.vu.lt/VU:ELABAETD192847232&prefLang=en_US