Title: Interactive crawling and data extraction Author: Bc. Petr Fejfar Author's e-mail address: pfejfar@gmail.com Department: Department of Distributed and Dependable Systems Supervisor: Mgr. Pavel Je ek, Ph.D., Department of Distributed and De- pendable Systems Abstract: The subject of this thesis is Web crawling and data extraction from Rich Internet Applications (RIA). The thesis starts with analysis of modern Web pages along with techniques used for crawling and data extraction. Based on this analysis, we designed a tool which crawls RIAs according to the instructions defined by the user via graphic interface. In contrast with other currently popular tools for RIAs, our solution is targeted at users with no programming experience, including business and analyst users. The designed solution itself is implemented in form of RIA, using the Web- Driver protocol to automate multiple browsers according to user-defined instructions. Our tool allows the user to inspect browser sessions by dis- playing pages that are being crawled simultaneously. This feature enables the user to troubleshoot the crawlers. The outcome of this thesis is a fully design and implemented tool enabling business user to extract data from the RIAs. This opens new opportunities for this type of user to collect data from Web pages for use...

This work deals with the automation of a web browser - the tools that allow programmatic control of the program for browsing the web pages. First, it discusses the existing solutions with focus on the tools from the Selenium Suite family and PhantomJS. Further, the internal representation of the web pages in the Gecko and WebKit browser engines is discussed. The work then focuses on the web browser application interface available for client-side scripting. The relevant standards are discussed as well. The core part of the thesis is dedicated to the design and implementation of a tool that allows to control a browser using the Selenium WebDriver tool and to extract data about the targert web page. The work presents an internal architecture, configuration files and the application interface of the designed tool. The topic of extracting detailed data about the page and its transformation to a unified structured description is covered as well. Finally, the performed unit tests and tests on real web pages are described.

This thesis deals with the web page segmentation in a web browser. The implementation of Box Clustering Segmentation (BCS) method in JavaScript using an automated browser was created. The actual implementation consists of two main steps, which are the box extraction (leaf DOM nodes) from the browser context and their subsequent clustering based on the similarity model defined in BCS. Main result of this thesis is a functional implementation of BCS method usable for web page segmentation. The evaluation of the functionality and accuracy of the implementation is based on a comparison with a reference implementation created in Java.

The validation approach for assuring quality of software does often include the conduction of tests. Software testing includes a wide range of methodology depending on the system level and the component under test. Graphical user interface (GUI) testing consists of high level tests that assert that functions and design element in user interfaces work as expected. The research conducted in this paper focused on GUI testing of web based applications and the movement towards automated testing within the software industry. The question which formed the basis for the study was the following: How should a generic test automation framework be designed in order to allow maintenance between developers and non-developers? The study was conducted on a Swedish consultant company that provides e-commerce web solutions. A work strategy approach for automated testing was identified and an automation framework prototype was produced. The framework was evaluated through a pilot study where testers participated through the creation of a test suite for a specific GUI testing area. Time estimations were collected as well as qualitative measurements through a follow up survey. This paper presents a work strategy proposal for automated tests together with description of the framework system design. The results are presented with a subsequent discussion about the benefits and complexity of creating and introducing automated tests within large scale systems. Future work suggestions are also addressed together with accountancy of the frameworks usefulness for other testing areas besides GUI testing.