Android is one of the most popular mobile operating systems on the market today with a vast majority of the market share when it comes to mobile devices. Graphical user interfaces (GUI) are often seen on the applications on these devices. Testing of these GUIs is important since they often make up for half of the source code of the application, and they are used to interact with the application. Automating these tests is very useful since it saves a lot of time, but can be difficult. The tools that are available for automating the tests are often not suitable for the developers’ needs, because of the lack of functionality. Therefore, the need for a characterization of the frameworks is needed, so that developers more easily can fit a framework for their needs. In this study, four open source frameworks for Android GUI testing have been selected for evaluation: Robotium, Selendroid, UI Automator and Espresso. Criteria used in the evaluation have been identified with the help of a literature analysis.The results show that two of the frameworks, Robotium and Espresso, lack the ability to fully test activities, which is the main component of Android application GUIs. Furthermore, the study resulted in characterizations of the frameworks.

The number of mobile applications has increased tremendously over the last decade, thereby also having increased the importance of mobile application testing. Testing is a very crucial process in the development of a mobile application to ensure reliability and proper functionality. In an Agile/DevOps environment, test automation is an integral part in order to support the continuous integration and continuous delivery/deployment principle. There is a plethora of mobile application testing automation frameworks available in the market today. Android testing frameworks were chosen for this study because of the prevalence of Android applications in the market today. The focus of this study is the comparison of the most popular mobile application testing automation frameworks in terms of performance and capabilities in an Agile/DevOps environment. In order to achieve this, firstly, the three most popular mobile application testing automation frameworks to be used for the study were identified. Secondly, the Key Performance Indicators and capabilities to be used for comparison of the frameworks were identified. Then, test cases in a complex mobile application were designed for analysing the frameworks based on the criteria gathered. Finally, the test cases were integrated in a Continuous Integration/ Continuous Delivery pipeline to gather results and perform a more detailed comparison in an Agile/DevOps environment. From this study, Espresso turned out to be the best framework in terms of KPI analysed beating the other frameworks in all the KPIs, while Robotium was second best followed by Appium. But on the other hand, Appium was the best framework in terms of capabilities by having ability to execute the maximum number of capabilities analysed followed by Robotium while Espresso was the worst framework in this criteria. / Antalet mobilapplikationer har ökat enormt under det senaste decenniet och därmed har även vikten av mobilapplikationstest ökat. Testning är en mycket viktig process i utvecklingen av en mobilapplikation för att säkerställa tillförlitlighet och korrekt funktionalitet. I en Agile/DevOps miljö är testautomation en viktig del för att stödja principen av kontinuerlig integration och kontinuerlig leverans/driftsättning. Det finns en mängd ramverk för automatiserad testning av mobilapplikationer på marknaden idag. Testramverk för Android valdes för denna studie på grund av utbredningen av Android-applikationer på marknaden idag. Fokus för denna studie är jämförelsen av de mest populära automatiserade testramverk för mobilapplikationer med avseende på prestanda och lämplighet i en Agile/DevOps miljö. För att uppnå detta, identifierades först de tre mest populära automatiserade testramverk för mobilapplikationer som skulle användas för studien. Sedan identifierades de viktigaste resultatindikatorerna och kapaciteterna som skulle användas för att jämföra ramverken. Därefter designades testfall i en komplex mobilapplikation för att analysera ramverken baserat på de bestämda kriterierna. Slutligen integrerades testfallen i en pipeline för kontinuerlig integration /kontinuerlig leverans för att samla resultat och utföra en mer detaljerad jämförelse i en Agile/DevOps miljö. Från denna studie visade sig Espresso vara det bästa ramverket när det gäller KPI-analys, som slog de andra ramarna i alla KPI: er, medan Robotium var näst bäst följt av Appium. Men å andra sidan var Appium det bästa ramverket med avseende på kapacitet. Den hade förmågan att utföra det maximala antalet kapacitet som analyserades, följt av Robotium medan Espresso var det sämsta ramverket i dessa kriterier.