Mobile application development and testing involves even greater challenges than traditional software development, given that the large number of existing devices, each with unique hardware configuration and operating system needs to be considered. Software testing constitutes a big and important part of development, where the majority of the tests are performed manually. This is time consuming and expensive. The purpose of this thesis project is therefor to examine the current test process at the company Dewire, in an effort to find ways to improve it by introducing automated tests for the Android mobile platform. The automation frameworks Appium and Espresso will be evaluated and combined with existing systems like Jenkins that already applied in Dewires agile work procedure. A test environment been implemented during this thesis project as a proof of concept, where tests are run on a real device as the application is being built. Test cases from an ongoing development project has been implemented and the results evaluated. This thesis project has shown that the framework Espresso together with a Jenkins build-server has great potential to reduce manual testing and streamline the test process. It has also shown that test automation should be implemented from start of a project, where tests are continuously kept up to date during development to be profitable. At the same time results show that test automation involves a significant time investment before time and money can be saved. / Mobilapplikationsutveckling och testning innebär en större utmaning än traditionell mjukvaruutveckling eftersom hänsyn måste tas till ett stort antal enheter, alla med unik hårdvara, skärmstorlek och operativsystem. Mjukvarutestning utgör en viktig och stor del av utvecklingen av applikationer där majoriteten av testerna idag genomförs manuellt. Detta är tidskrävande och ekonomiskt kostsamt. Syftet med detta examensarbete är därför att undersöka nuvarande testprocess hos företaget Dewire och hitta möjliga förbättringar genom att införa automatiserade tester för mobilplattformen Android. Automatiseringsverktygen Appium och Espresso utvärderas i kombination med befintliga system som Jenkins som redan används i Dewires agila arbetssätt. Under examensarbetet har en testmiljö implementerats som ett sätt för visa att konceptet fungerar, där tester körs på en testenhet i samband med att applikationen byggs. Testfall från pågående utvecklingsprojekt har implementerats och resultaten av denna automatisering har utvärderats. Examensarbetet har visat att verktyget Espresso tillsammans med en Jenkins byggserver har stor potential att effektivisera och minska manuellt testande. Arbetet visar också att testautomatisering bör implementeras från projektstart och att tester aktivt underhålls under utvecklingen för att vara lönsamt. Resultatet visar samtidigt att testautomatisering till en början innebär en väsentlig tidsinvestering innan tid och pengar kan sparas.

Testing is an important activity when developing a system. Testing requires resources in terms of time, labour and money. By correctly automating the tests, the development time may either be shortened or there will be a possibility to run more tests. ABB in Ludvika has developed MACH, a control & protection system for HVDC (high power electrical transmission over long distances) applications. During development of the control & protection system for each HVDC project, which are all unique, the system is today tested manually, which takes considerable time.This thesis project studies the possibility of automating parts of the MACH system tests, by investigating current testing procedures, the control & protection system itself, and how a test tool may interact with the system. Using this information a test framework, aimed towards test automation, was created, and a simple test execution tool was developed. A new test case, a combination of multiple smaller test cases, ranon the system using the test execution tool.The outcome proves the proof of concept of automating parts of the system tests.The economics and the scope of the automated testing however, is dependent on how automation is implemented.

Regression testing is just as important for the quality assurance of a system, as it is time consuming. Several techniques exist with the purpose of lowering the execution times of test suites and provide faster feedback to the developers, examples are ones based on transition-models or string-distances. These techniques are called test case selection (TCS) techniques, and focuses on selecting subsets of the test suite deemed relevant for the modifications made to the system under test. This thesis project focused on evaluating the use of a topic model, latent dirichlet allocation, as a means to create a diverse selection of test cases for coverage of certain test characteristics. The model was tested on authentic data sets from two different companies, where the results were compared against prior work where TCS was performed using similarity-based techniques. Also, the model was tuned and evaluated, using an algorithm based on differential evolution, to increase the model’s stability in terms of inferred topics and topic diversity. The results indicate that the use of the model for test case selection purposes was not as efficient as the other similarity-based selection techniques studied in work prior to thist hesis. In fact, the results show that the selection generated using the model performs similar, in terms of coverage, to a randomly selected subset of the test suite. Tuning of the model does not improve these results, in fact the tuned model performs worse than the other methods in most cases. However, the tuning process results in the model being more stable in terms of inferred latent topics and topic diversity. The performance of the model is believed to be strongly dependent on the characteristics of the underlying data used to train the model, putting emphasis on word frequencies and the overall sizes of the training documents, and implying that this would affect the words’ relevance scoring to the better.

Testing is a necessary component of software development, but it is also an expensive one, especially if performed manually. One way to mitigate the cost of testing is to implement test automation, where the test cases are run automatically. For any organisation looking to implement test automation, the most interesting cost is time. Automation takes time to implement and one of the most obvious benefits of automation is that the automated test execution time is lower than that of manual execution. This thesis contains a literature study covering testing methodology, especially in regards to the domain of web application testing. The literature covered also included three economic models that may be used to calculate the costs of automation compared to manual testing. The models can be used to calculate the time it would take, or the number of necessary executions, for the total cost of test automation to be lower than of that of manual testing. The thesis is based on a case study of test automation for the StoredSafe platform, a web application. Three sets of test automation frameworks were used to implement three different test suits and the test implementation times were collected. The data collected were then used to calculate the time it would take, using the three economic models, for the cost of automated test cases to become equal to that of with manual testing. The data showed that the estimated time to reach breakeven for the three frameworks varied between 2½ and at worst 10 years, with an average of 3½ years. The models and data presented in this thesis may be used in order to estimate the cost of test automation in comparison to manual testing over longer periods of time, but care must be taken in order to ensure that the data used is correct in regards to one’s own organisation or else the estimate may be faulty.

I takt med ökad användning av olika slags informationssystem växer även behovet av att testa mjukvaran, vilket innebär att allt fler vill komma igång med testautomation. Denna studie syftar till att beskriva hur man går tillväga för att ta första steget mot testautomationen med hjälp av Robot Framework, samt betona vikten av kravspecifikationer. Studien är en Design Science Research (designbaserad forskning) som löser ett verkligt problem på ett verkligt företag som i dagsläget anställer en hel avdelning av manuella testare och resulterar i en artefakt vilkens fullständig arkitektur redovisas i denna rapport. Problemet går ut på att automatisera ett testflöde, det vill säga en samling av tester som utförs manuellt. Testflödet behandlar den mest komplexa komponenten i hela systemet och tar en hel arbetsdag för manuella testare. Efter utförlig analys och automation förkortas testningstiden till tjugo minuter. Slutprodukten är en enkel applikation som guidar användaren igenom testet där användaren får välja vilka tester ska exekveras, samt sparar exekveringsresultat bestående av loggfiler och (om så önskats) en skärminspelning i en zip-fil. / As the usage of various information systems increases, so does the need to test software, meaning that more companies are interested in getting started with test automation. This study aims to describe how to proceed in order to take the first step towards test automation using Robot Framework, as well as to emphasize the importance of requirement specifications. The study is a Design Science Research, which is solving a real-life problem in a real company that currently employs a whole department of manual testers and is leading to the development of an artifact. The complete architecture of the artifact is disclosed in this report. The problem involves automating a test flow, that is, a collection of tests that are executed manually. The test is carried out on the most complex component in the entire system and takes a whole workday for manual testers. After thorough analysis and automation, the testing time is shortened to twenty minutes. The end product is a simple application that guides the user through the test where the user gets to choose which tests shall be executed, and saves the execution results consisting of log files and (if so wished) a screen recording in a zip file.