This report is the proof of concept that shows how Coded UI can be used for the automation of GUI tests. For this thesis-work, an industrial case study is done on 26 different test cases provided at the company Basalt AB. The problem they had was that testing was very time consuming and was therefore not done as often as needed to keep the level of quality required for the system developed. The method that has been used are called the validation method and the work was according to that divided into five steps: choosing test cases, a learning process, implementation, compare the results and then draw conclusions. Test automation has benefits of time savings and the fact that tests will be reused. Testing will take minutes instead of hours and the tests can be executed over and over again as many times as needed. So by changing from manual testing to automated testing, tests can be run faster and therefore more often. The investment of automated testing will be repaid before the end of the fourth test round. So conclusions can be drawn from the results of this thesis and it shows that it absolutely is worth the investment of automating the testing process.

Efficiency as a term in software testing is, in the research community, a term that is not so well defined. In the industry, and specifically the test tool industry, it has become a sales pitch without meaning. GUI testing in its manual form is a time consuming task, which can be thought of as repetitive and tedious by testers. Using human testers to perform a task, where focus is hard to keep, often ends in defects going unnoticed. The purpose of this thesis is to collect knowledge on the area efficiency in software testing, but focusing more on efficiency in GUI testing in order to keep the scope focused. Part of the purpose is also to test the hypothesis that automated GUI testing is more efficient than traditional, manual GUI testing. In order to reach the purpose, the choice fell to use case study research as the main research method. Through the case study, a theoretical study was performed to gain knowledge on the subject. To gain data used for an analysis in the case study, the choice fell on using a semi-experimental research approach where one automated GUI testing technique called Capture & Replay was tested against a more traditional approach towards GUI testing. The results obtained throughout the case study gives a definition on efficiency in software testing, as well as three measurements on efficiency, those being defect detection, repeatability of test cases, and time spent with human interaction. The result also includes the findings from the semi-experimental research approach where the testing tools Squish, and TestComplete, where used beside a manual testing approach. The main conclusion deducted in this work is that an automated approach towards GUI testing can become more efficient than a manual approach, in the long run. This is when efficiency is determined on the points of defect detection, repeatability, and time.

Background. Speed is getting more and more critical in modern Software Engineering to be able to respond to users’ expectations of product development. One practice that takes a significant amount of time in the process of releasing software to the customers is testing. It is a clear trend that organizations are increasing the amount of test automation compared to manual testing. However, manual testing is still prominent in GUI-based testing due to challenges interacting with a GUI from test scripts. One of the most prominent challenges is synchronizing test script execution with the system under test. Objectives. This research aims to compare static and dynamic synchronization of GUI-based tests. This comparison will be conducted by replicating an existing Selenium test suite using static synchronization to Playwright, which will use dynamic synchronization and run the test suites against a web-based application. These test suites will then be used to compare the two types of synchronization from the perspective of test execution efficiency, test output correctness, and maintenance cost.Methods. The research methodology we chose is experiment. We have chosen Selenium to represent static synchronization and Playwright to represent dynamic synchronization. We used an existing test suite in Selenium that was translated into Playwright. There are a total of 81 tests in the test suite. The test suites were used to compare test scripts that use static and dynamic synchronization from the perspective of test execution efficiency, test output correctness, and maintenance cost.Results. The data collected from the experiments shows that execution efficiency for test scripts using dynamic synchronization is significantly faster than static synchronization. A mean difference between the test suites showed a decrease of 87%. For defect identification, ten defects where used and both test suites managed to identify all of them, resulting in no difference could be found. Test maintenance cost was found that test scripts using dynamic synchronization had a positive effect with an average of 60% less time spent on maintenance.Conclusions. Based on the result, we found that test scripts using dynamic synchronization improved execution efficiency and maintenance costs without sacrificing the test output correctness.

Background. With the development and popularization of mobile Internet, smartphones are becoming more and more popular. Android is one of the most popular platforms of smartphones.  And application is one of the most important part of a smartphone. There are a lot of money and human resources spent on Android application development every year. And quiet a big part of them goes to quality assurance of applications. Graphic user interface (GUI) testing is one important part of its quality assurance. Android phones use touch screen as the major I/O method. Therefore, GUI testing on android platform shall be different to conventional software applications that are designed to run on desktop environment. Objectives. The aim of this research is to assess the performance of two GUI testing approaches (2nd vs 3rd generation) of automated UI testing in terms of testing Android applications. By assessing these approaches, we could hopefully get insights of their advantages and limitations for using them in the context of Android development. And this aim can be divided into three objectives, to compare the time spent on implementing test cases of each tool, to compare the time costed when executing test cases of each tool, to compare the number of defects found by each tool. Methods. The research methodology we chose is controlled experiment. We have chosen UI Automator and Appium to represent 2nd generation GUI testing approach, EyeAutomate and SikuliX to represent 3rd generation GUI testing approach. We used each tool to implement and execute 120 test cases to compare them on the time spent on implementing test cases of each tool, the time costed when executing test cases of each tool, the number of real defects found by each tool, and the number of false positives found by each tool. Results. Tools using 3rd generation GUI testing approach take less time to implement test cases than tools using 2nd generation GUI testing approach. And there is no specific pattern when comparing tools using 2nd and 3rd generation GUI testing approaches in terms of time cost on executing test cases. It is different between different test cases. Besides false positive alerts appear at a much higher frequency in tools using 3rd generation GUI testing approach than tools using 2nd generation GUI testing approach. While, real defects found by each tool are the same. Conclusions. 3rd generation GUI testing approach is more efficient in terms of implementing test cases than 2nd generation GUI testing approach. But 3rd generation GUI testing approach finds much more false positives than 2nd generation approach. To decide if a defect alert is false positive or not requires human effort. In a long term, it may accumulate huge lost on human efforts. Therefore, to maintain test cases, 3rd generation approach consumes lots of human efforts.

Made available in DSpace on 2014-06-12T15:54:24Z (GMT). No. of bitstreams: 2 arquivo7096_1.pdf: 2072025 bytes, checksum: ca8b71b9cfdeb09118a7c281cafe2872 (MD5) license.txt: 1748 bytes, checksum: 8a4605be74aa9ea9d79846c1fba20a33 (MD5) Previous issue date: 2010 / Conselho Nacional de Desenvolvimento Científico e Tecnológico / Aplicações para celular estão se tornando cada vez mais complexas, bem como testá-las. Teste de interfaces gráficas (GUI) é uma tendência atual e se faz, em geral, através da simulação de interações do usuário. Várias técnicas são propostas, no qual, eficiência (custo de execução) e eficácia (possibilidade de encontrar bugs) são os aspectosmais cruciais desejados pela industria. No entanto, avaliações mais sistemáticas são necessárias para identificar quais técnicas melhoram a eficiência e eficácia de tais aplicações. Esta tese apresenta uma avaliação experimental de duas técnicas de testes de GUI, denominadas de DH e BxT, que são usadas para testar aplicações de celulares com um histórico de erros reais. Estas técnicas são executadas por um longo período de tempo (timeout de 40h, por exemplo) tentando identificar as situações críticas que levam o sistema a uma situação inesperada, onde o sistema pode não continuar sua execução normal. Essa situação é chamada de estado de crash. A técnicaDHjá existia e é utilizadapela industriade software, propomos outra chamada de BxT. Em uma avaliação preliminar, comparamos eficácia e eficiência entre DH e BxT através de uma análise descritiva. Demonstramos que uma exploração sistemática, realizada pela BxT, é uma abordagem mais interessante para detectar falhas em aplicativos de celulares. Com base nos resultados preliminares, planejamos e executamos um experimento controlado para obter evidência estatística sobre sua eficiência e eficácia. Como ambas as técnicas são limitadas por um timeout de 40h, o experimento controlado apresenta resultados parciais e, portanto, realizamos uma investigação mais aprofundada através da análise de sobrevivência. Tal análise permite encontrar a probabilidade de crash de uma aplicação usando tanto DH quanto BxT. Como experimentos controlados são onerosos, propomos uma estratégia baseada em experimentos computacionais utilizando a linguagem PRISM e seu verificador de modelos para poder comparar técnicas de teste de GUI, em geral, e DH e BxT em particular. No entanto, os resultados para DH e BxT tem uma limitação: a precisão do modelo não é estatisticamente comprovada. Assim, propomos uma estratégia que consiste em utilizar os resultados anteriores da análise de sobrevivência para calibrar nossos modelos. Finalmente, utilizamos esta estratégia, já com os modelos calibrados, para avaliar uma nova técnica de teste de GUI chamada Hybrid-BxT (ou simplesmente H-BxT), que é uma combinação de DH e BxT

Android is one of the most popular mobile operating systems on the market today with a vast majority of the market share when it comes to mobile devices. Graphical user interfaces (GUI) are often seen on the applications on these devices. Testing of these GUIs is important since they often make up for half of the source code of the application, and they are used to interact with the application. Automating these tests is very useful since it saves a lot of time, but can be difficult. The tools that are available for automating the tests are often not suitable for the developers’ needs, because of the lack of functionality. Therefore, the need for a characterization of the frameworks is needed, so that developers more easily can fit a framework for their needs. In this study, four open source frameworks for Android GUI testing have been selected for evaluation: Robotium, Selendroid, UI Automator and Espresso. Criteria used in the evaluation have been identified with the help of a literature analysis.The results show that two of the frameworks, Robotium and Espresso, lack the ability to fully test activities, which is the main component of Android application GUIs. Furthermore, the study resulted in characterizations of the frameworks.

GUI testing is expensive to perform manually. Software systems involving a heterogeneous set of components exclude the applicability of specific GUI testing techniques. Visual GUI Testing (VGT) is a test automation technique that combines image recognition with scripts. It applies to almost any GUI driven application. VGT is proven to be cost-effective in comparison with manual testing. Still, it is expensive to maintain. This study investigates if test script design by following specific guidelines positively affects maintenance costs. A case study was conducted to identify best practices for VGT w.r.t. maintenance time. Four VGT versions were developed for each manual test case. These consisted of two design versions, with/without guidelines, for the two VGT-tools EyeAutomate and Sikuli. Data was collected using time measurements, observations, and interviews. Results highlighted differences in initial development time and maintenance time between the two design versions. In total, 44 observations were collected. 17 were related to the design versions, 17 to the VGT-tools, and 10 to VGT in general, initial development, and the system under test. The interviews collected the perceptions of VGT in general, maintenance of the different VGT versions, and guidelines. In conclusion, the combination of the guidelines did not have a positive effect on maintenance in terms of costs and experience. However, some of the individual guidelines did. A rationale why the guidelines did not give the desired result was identified. Future research is necessary to investigate other combinations of guidelines, such as those identified as beneficial.

Graphical user interfaces represent significant part of computer systems. This causes an increase of demands for their proper behavior, which can be achieved by a proper testing during development. This thesis describes the principles of model based testing for automated generation of test cases. In this way, the productivity of test teams can increase. The thesis defines model for modeling the most common GUI elements, algorithm for obtaining such a model from the GUI, and a tool for definition of coverage criteria.

The diploma thesis deals with regression testing of the Campaign Manager web application of the Xtend product. The work interprets the software development methodologies and subsequently it describes the types of testing that are closely related to development. Furthermore, the work focuses on tools used for the graphical user interface testing and their mutual comparison. The next part of the work is dedicated to the creation of the development environment and it analyzes the structure of test cases that were developed in the created environment. The last section describes how the Campaign Manager was tested and how test results were generated and obtained.

This thesis describes a system for automated GUI testing using assistive technologies for accessing and manipulating GUI elements. The only input from the user to automated test system is a description of UI events and activities. For this purpose, a specialized language is proposed. The test system then automatically generates possible sequences of UI events applying a given criterion. Generated test set is executed by Python interpreter exploiting the Linux Desktop Testing Project (LDTP). Test system described in this thesis then provides reports and coverage evaluation for particular test cases and the whole test set.