Efficiency as a term in software testing is, in the research community, a term that is not so well defined. In the industry, and specifically the test tool industry, it has become a sales pitch without meaning. GUI testing in its manual form is a time consuming task, which can be thought of as repetitive and tedious by testers. Using human testers to perform a task, where focus is hard to keep, often ends in defects going unnoticed. The purpose of this thesis is to collect knowledge on the area efficiency in software testing, but focusing more on efficiency in GUI testing in order to keep the scope focused. Part of the purpose is also to test the hypothesis that automated GUI testing is more efficient than traditional, manual GUI testing. In order to reach the purpose, the choice fell to use case study research as the main research method. Through the case study, a theoretical study was performed to gain knowledge on the subject. To gain data used for an analysis in the case study, the choice fell on using a semi-experimental research approach where one automated GUI testing technique called Capture & Replay was tested against a more traditional approach towards GUI testing. The results obtained throughout the case study gives a definition on efficiency in software testing, as well as three measurements on efficiency, those being defect detection, repeatability of test cases, and time spent with human interaction. The result also includes the findings from the semi-experimental research approach where the testing tools Squish, and TestComplete, where used beside a manual testing approach. The main conclusion deducted in this work is that an automated approach towards GUI testing can become more efficient than a manual approach, in the long run. This is when efficiency is determined on the points of defect detection, repeatability, and time.

This thesis describes a system for automated GUI testing using assistive technologies for accessing and manipulating GUI elements. The only input from the user to automated test system is a description of UI events and activities. For this purpose, a specialized language is proposed. The test system then automatically generates possible sequences of UI events applying a given criterion. Generated test set is executed by Python interpreter exploiting the Linux Desktop Testing Project (LDTP). Test system described in this thesis then provides reports and coverage evaluation for particular test cases and the whole test set.

The effectiveness of GUI-based automated tests is a topic that is often discussed in the testing community. As software systems and GUIs are getting more advanced, the testing of these systems may sometimes be both time-consuming and repetitive. Therefore, automated tests are used more frequently to create more agile working and to decrease the number of manual tests. There is an ongoing discussion about the benefits and shortcomings of implementing automated GUI test compared to performing manual GUI tests. Increasingly, companies today are implementing automated GUI tests, yet relatively few are analyzing enough when choosing their test strategy. The purpose of this thesis was to explore different test strategies in a system as complex as Nasdaq’s clearing system NRTC. The first strategy used manual GUI tests, while the second strategy used a test tool called Selenium WebDriver to automate the GUI tests. Hopefully, this research will contribute to the already accumulated knowledge about automated GUI tests and potentially help in choosing the right testing strategy. The findings of this research showed that the implementation time of manual tests was much faster, and that implementing automated test would be timeconsuming. The reason for this is that the GUI was not very testable, making it difficult to create the test scripts. In addition, the was a difference in runtime when using both strategies. The automated tests only took a couple of seconds to execute, compared to the manual tests that took a couple of minutes. After both strategies was investigated, an analysis was done by looking at the effectivity, maintainability, robustness, and the overall results to be able to find benefits and shortcomings. In conclusion, GUI automation cannot replace manual test completely. By not including the human capabilities, the testing process becomes very limited.

Naujuose programiniuose produktuose plačiai taikoma mobilioji įranga (mobilieji telefonai, delniniai kompiuteriai). Programinės įrangos kūrimo procese svarbią vietą užima testavimas. Dabartinių mobiliųjų įranginių apribojimai (darbo sparta, atminties kiekis, energija, ekrano dydis, platformų įvairumas) kelia naujas problemas programinės įrangos kūrimo procesui, tame tarpe ir testavimui. Testavimo proceso automatizavimas leidžia sumažinti bandymų trukmę, padidinti testavimo darbų apimtį. Programinės įrangos testavime mobiliems įrenginiams svarbu išskirti grafinės vartotojo sąsajos testavimą. Svarbi testavimo dalis – testavimas pagal specifikaciją. Šiame darbe nagrinėjamas grafinės vartotojo sąsajos automatizuotas testavimas mobiliai įrangai, nustatomi tikslai, galimi keliai jiems pasiekti bei galimas testavimo aplinkos praplėtimas testų generavimui pagal specifikaciją. / Mobile devices such as cell phones and personal digital assistance are widely used in new software products. Testing takes important place in software development process. Constraints of mobile devices (speed, amount of memory, energy, small screen, wide range of platforms) raise new problems for software development process including testing phase. Automated approach of software testing reduces testing time and increases testing range. It is important to distinguish graphic user interface as a special part of testing. The important part of testing is specification based testing. The aim of these master theses is to analyze automated testing of GUI for mobile devices, define testing tasks and enhance mobile device testing framework by providing means for test case generation from specification.

Mobile applications are growing in popularity and pose new problems in the area of software testing. In particular, mobile applications heavily depend upon user interactions and a dynamically changing environment of system events. In this thesis, we focus on user-driven events and use Q-learning, a reinforcement machine learning algorithm, to generate tests for Android applications under test (AUT). We implement a framework that automates the generation of GUI test cases by using our Q-learning approach and compare it to a uniform random (UR) implementation. A novel feature of our approach is that we generate user-driven event sequences through the GUI, without the source code or the model of the AUT. Hence, considerable amount of cost and time are saved by avoiding the need for model generation for generating the tests. Our results show that the systematic path exploration used by Q-learning results in higher average code coverage in comparison to the uniform random approach.

Testning är en nödvändig men kostsam del av mjukvaruutveckling. Test utförs på olika abstraktionsnivåer och kan vara manuella eller automatiserade. På lägsta abstraktionsnivå, enhetsnivå, är automatiserad testning vanligt och relativt okomplicerat, medan systemtester är svårare att automatisera. I synnerhet gäller detta tester på ett grafiskt användargränssnitt (GUI) som kräver speciella verktyg.   Triona vill undersöka möjligheterna att automatisera regressionstester från GUI:t av sin produkt C-Load, en molnbaserad webbtjänst för avtalsbaserad transportbokning.    Det primära syftet med denna fallstudie är att med en anpassad urvalsprocess utvärdera ett möjligt verktyg i förhållande till C-Load-förvaltningens förväntningar på automatiserad GUI testning (AGT) och att utifrån resultatet föreslå hur C-Loadförvaltningen kan gå vidare med val av verktyg för AGT. För att uppfylla syftet användes litteraturstudier, intervjuer och observationer av praktiska test.    Verktyg för GUI-testning kan delas in i tre huvudkategorier: skriptbaserade, modellbaserade och skriptlösa. Baserat på tidigare forskning drogs slutsatsen att ett skriptbaserat verktyg där koden i testskripten skrivs manuell bäst passar C-Loadförvaltningens krav och förutsättningar. Det mest använda verktyget av denna typ, Selenium WebDriver, utvärderades kvalitativt gentemot identifierade krav.    Av tidigare forskning framgår att vanliga utmaningar med skriptbaserade GUI-tester är att arbetsinsatsen för att skapa och underhålla testskript är stor och att testen kan vara opålitliga. Dessa problem framkom också i studiens intervjuer och observationer.   Slutsatsen är att det vore möjligt att automatisera regressionstester av C-Load med hjälp av Selenium Webdriver, och att det på sikt skulle kunna frigöra tid. Initialt krävs dock en omfattande insats för att implementera automatiserade tester i förvaltningen och Selenium Webdriver uppfyller bara delvis C-Load-förvaltningens förväntningar på AGT. C-Load-förvaltningen rekommenderas att utvärdera fler verktyg innan beslut fattas. I en kommande urvalsprocess bör Triona beakta hur väl olika verktyg fungerar i förhållande till moderna webbramverk. / Testing is a necessary but costly part of software development. Tests are performed at different abstraction levels and can be either manual or automated. On the lowest level of abstraction, where unit testing is performed, automated testing is commonplace and relatively uncomplicated, whereas system testing is more difficult to automate. This is especially true for GUI-testing, which requires special tools.      Triona wished to investigate possibilities to automate regression testing of the GUI for its C-load product, which is a Cloud-based web-service for contract-based transport booking.      The purpose of this case study was to evaluate one tool for automated GUI-testing (AGT) against the C-Load team’s expectations on AGT, and based on the result recommend Triona how to proceed in the process of implementing AGT. Literature studies, observations and interviews were conducted to fulfil the purpose.      GUI-testing tools can be classified into three categories: script-based, model-based and scriptless. One conclusion was that a script-based tool, where test scripts are manually coded would best suit Triona’s needs. The most used tool in that category, Selenium WebDriver, was tested and evaluated against requirements.       Prior research shows that common challenges encountered when using script-based GUItests are the workload required to create and maintain test scripts and that the tests can deliver inconsistent or “flaky” results. These challenges were confirmed during our analysis.       Our conclusion is that it is possible to automate C-Load regression tests with Selenium WebDriver, and that it would eventually free up time. However, a considerable effort is initially required to implement automated testing. Selenium Webdriver only partly fulfills the C-Load team’s expectations on AGT. Before a decision is taken, the C-Load team should evaluate more tools. When evaluating tools for AGT, Triona should take note that Selenium Webdriver can be deficient when it comes to testing applications based on modern web frameworks.