<p>Within Ericsson AB, integration and verification activities is done on the network level in order to secure the functionality of the network. Protocol analysers are used to capture the traffic in the network. This results in many log files, which needs to be analysed. To do this, a protocol decoder tool called Scapy/LHC is used. Scapy/LHC is a framework that allows the users to write their own script to retrieve the data they need from the log files. The Scapy/LHC framework is incrementally developed as open source within Ericsson when there are needs for more functionality. This is often done by the users, outside normal working tasks. Because of this, there is almost no testing done to verify that old and new functionality works as expected, and there is no formal test strategy in use today.</p><p><br />The goal of this master’s thesis is to evaluate test strategies that are possible to use on the Scapy/LHC framework. To make the time needed for the testing process as short as possible, the test strategy needs to be automated. Therefore, possible test automation tools shall also be evaluated.</p><p><br />Two possible test strategies and two possible test automation tools are evaluated in this thesis. A test strategy, where the scripts that are written by the users are used, is then selected for implementation. The two test automation tools are also implemented. The evaluation of the implemented test strategy shows that it is possible to find defects in the Scapy/LHC framework in a time efficient way with help of the implemented test strategy and any of the implemented test automation tools.</p>

More and more companies experience problems with maintainability and time-consuming development of automated testing tools. The MPC department at Ericsson Software Technology AB use methods and tools often developed during time pressure that results in time-consuming testing and requires more effort and resources than planned. The tools are also such nature that they are hard to expand, maintain and in some cases they have been thrown out between releases. For this reason, we could identify two major objectives that MPC want to achieve; efficient and maintainable test automation. Efficient test automation is related to mainly how to perform tests with less effort, or in a shorter time. Maintainable test automation aims to keep tests up to date with the software. In order to decide how to achieve these objectives, we decided to investigate which test to automate, what should be improved in the testing process, what techniques to use, and finally whether or not the use of automated testing can reduce the cost of testing. These issues will be discussed in this paper.

Within Ericsson AB, integration and verification activities is done on the network level in order to secure the functionality of the network. Protocol analysers are used to capture the traffic in the network. This results in many log files, which needs to be analysed. To do this, a protocol decoder tool called Scapy/LHC is used. Scapy/LHC is a framework that allows the users to write their own script to retrieve the data they need from the log files. The Scapy/LHC framework is incrementally developed as open source within Ericsson when there are needs for more functionality. This is often done by the users, outside normal working tasks. Because of this, there is almost no testing done to verify that old and new functionality works as expected, and there is no formal test strategy in use today. <br />The goal of this master’s thesis is to evaluate test strategies that are possible to use on the Scapy/LHC framework. To make the time needed for the testing process as short as possible, the test strategy needs to be automated. Therefore, possible test automation tools shall also be evaluated. <br />Two possible test strategies and two possible test automation tools are evaluated in this thesis. A test strategy, where the scripts that are written by the users are used, is then selected for implementation. The two test automation tools are also implemented. The evaluation of the implemented test strategy shows that it is possible to find defects in the Scapy/LHC framework in a time efficient way with help of the implemented test strategy and any of the implemented test automation tools.

Ericsson has a very big and expensive test environment with a lot of GSM AXE equipment. In order to decrease the cost of testing Ericsson has developed a combination of simulated and real hardware, the Hybrid Test Environment (HTE). There is no formal supervision and testing of the HTE system today and this has left the HTE system unstable and the testers have been avoiding using HTE. It is important for Ericsson that the confidence for HTE will increase. The goal of this thesis is to produce a method for testing the HTE system. An automated test tool has been implemented in order to monitor and test the HTE system. During the two weeks that the test tool has been operational it has discovered 4 servers in 3 different HTE rigs that malfunctioned. These servers were fixed and were operational before the end-users could discover any problem.

In the modern wireless communication industry, radio communication equipment vendors not only produce communication hardware, but also produce software. In fact, software revenue is now a large part of the total revenue. As technology has developed and traffic demands increase, more and more functions required to implement the radio system are implemented via software rather than hardware. Today, many hardware functions are actually implemented with reconfigurable and programmable hardware. Therefore, it is often possible to perform an upgrade by loading new software (a software upgrade) rather than needing to change the physical hardware with every technology advance. However, introducing new elements and features in existing (often mature) software may cause unexpected problems. These problems may include new parts malfunctioning and failure or degradation of old functions. To avoid these problems, each version of software has to be thoroughly tested, not only to test the new parts, but also to verify that the old functions still work properly. Testing all the old functions is time and human resource consuming. Thus, there is an increasing demand for automated testing. This thesis will focus on why automated regression testing is necessary and how to implement automated testing in a specific environment. The thesis results show that automated testing can improve the test coverage by at least 40% for one of Ericsson’s WCDMA software releases. This coupled with a reduction in testing time enables more rapid development by significantly reducing the test time without compromising quality. All of these results lead to improved profitability and increased customer satisfaction. / I den moderna trådlösa kommunikationen industrin, radioutrustning leverantörerna inte bara producera kommunikation hårdvara, utan också producera mjukvara. Faktum är programvara inkomster är nu en stor del av de totala inkomsterna. Eftersom tekniken har utvecklat och trafik krav ökar, fler och fler funktioner som krävs att genomföra radiosystem genomförs via mjukvara istället för maskinvara. Många hårdvara fungerar faktiskt genomförs med omkonfigurerbara och programmerbar hårdvara. Därför är det ofta möjligt att utföra en uppgradering av lastning ny programvara (en mjukvaruversionen) snarare än behöver för att ändra den fysiska hårdvaran med varje teknik förväg. Men att införa nya element och funktioner i befintliga (ofta äldre) programvara kan orsaka oväntade problem. Dessa problem kan innehålla nya delar brister och fel eller försämring av gamla funktioner. För att undvika dessa problem, varje version av programvaran måste testas, inte bara att testa de nya delarna, men även för att kontrollera att de gamla funktionerna fortfarande arbete ordentligt. Testa alla gamla funktioner konsumera tid och personal. Således finns det en ökad efterfrågan på automatiserade tester. Den här avhandlingen kommer att fokusera på varför automatiserad regression testning krävs och hur man genomföra automatiserade tester i en viss miljö. Avhandlingen visar att automatiserade tester kan förbättra testbunt täckning med minst 40% för ett av Ericssons WCDMA programversionerna. Detta i kombination med en minskning av provning tid möjliggör en snabbare utveckling av avsevärt minska test tid utan att kompromissa med kvaliteten. Alla dessa resultat leda till bättre lönsamhet och ökat kundvärde belåtenhet.

At Marginalen Bank, there are several microservices containing endpoints that are covered bytest automation. The documentation of which microservices and endpoints that are covered byautomated tests is currently done manually and is proven to contain mistakes. In the documentation, the test coverage for all microservices together and for every individual microserviceis presented. Marginalen Bank needs a way to automate this process with a system that cantake care of test coverage documentation and present the calculated data. Therefore, the purpose of this research is to find a way to create a real-time auto-test monitoring system thatautomatically detects and monitors microservices, endpoints, and test automation to documentand present test automation coverage on a website. The system is required to daily detect andupdate the documentation to be accurate and regularly find eventual changes. The implemented system that detects and documents the test automation coverage is calledTest Autobahn. For the system to detect all microservices, a custom hosted service was implemented that registers microservices. All microservices with the custom hosted service installedand extended to registers to Test Autobahn when deployed on a server. For the system todetect all endpoints of each microservice, a custom middleware was implemented that exposesall endpoints of a microservice with it installed. For the microservices to be able to install theseand get registered, a NuGet package containing the custom hosted service and the custom middleware, was created. To detect test automations, custom attributes models were created thatare supposed to be inserted into each test automation project. The custom attributes are placedin every test class and method within a project, to mark which microservice and endpoint thatis being tested within every automated test. The attributes of a project can be read throughthe assembly. To read the custom attributes within every test automation project, a consoleapplication, called Test Autobahn Automation Detector (TAAD), was implemented. TAADreads the assembly to detect the test automations and sends them to Test Autobahn. Test Autobahn couples the found test automation to the corresponding microservices and endpoints.TAAD is installed and ran on the build pipeline in Azure DevOps for each test automationproject to register the test automations. To daily detect and update the documentation of the test coverage, Quartz.NET hosted serviceis used. With Quartz.NET implemented, Test Autobahn can execute a specified job on a schedule. Within the job, Test Autobahn detects microservices and endpoints and calculates the testautomation coverage for the detection. The calculation of the test coverage from the latestdetection is presented on the webpage, containing both the test coverage for all microservicestogether and the test coverage for each microservice. According to the evaluations, the systemseems to function as anticipated, and the documentation is displaying the expected data.

Det finns en omfattande litteratur kring modellbaserad testning (MBT) men med få tecken på att metoden har fått något större genomslag i industrin. Målet med studien är att identifiera exempel där MBT används som testmetod inom industrin och eventuella hinder som finns i MBT-processen. För att genomföra detta används en mixed-methods ansats bestående av en systematisk litteraturstudie följt av en utforskande fallstudie. I fallstudien tillämpas MBT med hjälp av verktyget Modbat på ett mjukvarusystem. Endast ett fåtal industriella tillämpningar av MBT identifieras i litteraturstudien. Totalt sju studier kvarstår efter fulltextgranskningen. Studierna finns primärt inom mjukvaruindustrin och flygindustrin men innehåller även exempel från hälso- sjukvård och bilindustrin. Den utforskande fallstudien indikerar tre typer av hinder. Det första är mängden arbete med, samt bristande användarvänlighet hos verktygen. Den andra är svårigheten med att skriva ett adaptionslager som integrerar systemet med verktyget och modellen för att göra testfallen körbara. Det sista hindret är det kraftiga beroendet på att modellen utformas korrekt och stämmer med systemets tilltänkta beteende. Dessa tre hinder pekas även ut i verken från litteraturstudien. Vidare pekas bland annat även icke-tekniska svårigheter ut under litteraturstudien i form av att hela arbetsgruppen och ledningen behöver engageras för att införa ett nytt arbetssätt. Med en begränsad fallstudie och ett enkelt system bekräftas tre hinder i MBT-processen som även identifieras i litteraturgenomgången. MBT framstår som ett primärt akademiskt område med ett fåtal exempel på användning inom industrin. / There is extensive literature concerning model-based testing (MBT) but few signs that the method have had any major breakthrough in the industry. The goal of this study is to identify examples of MBT being used in the industry and any challenges faced during the MBT-process. The study is conducted using a mixed methods approach, consisting of a systematic literature review followed by an exploratory case-study. The case-study applies MBT to a software system using the MBT-tool Modbat.Only seven studies remain after the fulltext review is performed. The studies are primarily from the software and aerospace industries but also include examples from the healthcare and automotive industries. The exploratory case-study identifies three challenges. The first one is the amount of work and lacking usability related to the MBT-tools. The second challenge is implementing the adaption layer, integrating the system under test with the tool and model to make test cases executable. The final challenge is the dependency on a correct model representing the systems expected behaviour. These three challenges are also identified in the systematic literature review. Other challenges from the literature review include non-technical difficulties concerning training and the need to motivate staff and management.Using a limited case-study and a simple system three challenges, which are also identified in the literature review, throughout the MBT process are confirmed. MBT appears primarily as an academic subject with some examples of use in the industry.

Det finns en omfattande litteratur kring modellbaserad testning (MBT) men med få tecken på att metoden har fått något större genomslag i industrin. Målet med studien är att identifiera exempel där MBT används som testmetod inom industrin och eventuella hinder som finns i MBT-processen. För att genomföra detta används en mixed-methods ansats bestående av en systematisk litteraturstudie följt av en utforskande fallstudie. I fallstudien tillämpas MBT med hjälp av verktyget Modbat på ett mjukvarusystem. Endast ett fåtal industriella tillämpningar av MBT identifieras i litteraturstudien. Totalt sju studier kvarstår efter fulltextgranskningen. Studierna finns primärt inom mjukvaruindustrin och flygindustrin men innehåller även exempel från hälso- sjukvård och bilindustrin. Den utforskande fallstudien indikerar tre typer av hinder. Det första är mängden arbete med, samt bristande användarvänlighet hos verktygen. Den andra är svårigheten med att skriva ett adaptionslager som integrerar systemet med verktyget och modellen för att göra testfallen körbara. Det sista hindret är det kraftiga beroendet på att modellen utformas korrekt och stämmer med systemets tilltänkta beteende. Dessa tre hinder pekas även ut i verken från litteraturstudien. Vidare pekas bland annat även icke-tekniska svårigheter ut under litteraturstudien i form av att hela arbetsgruppen och ledningen behöver engageras för att införa ett nytt arbetssätt. Med en begränsad fallstudie och ett enkelt system bekräftas tre hinder i MBT-processen som även identifieras i litteraturgenomgången. MBT framstår som ett primärt akademiskt område med ett fåtal exempel på användning inom industrin. / There is extensive literature concerning model-based testing (MBT) but few signs that the method have had any major breakthrough in the industry. The goal of this study is to identify examples of MBT being used in the industry and any challenges faced during the MBT-process. The study is conducted using a mixed methods approach, consisting of a systematic literature review followed by an exploratory case-study. The case-study applies MBT to a software system using the MBT-tool Modbat.Only seven studies remain after the fulltext review is performed. The studies are primarily from the software and aerospace industries but also include examples from the healthcare and automotive industries. The exploratory case-study identifies three challenges. The first one is the amount of work and lacking usability related to the MBT-tools. The second challenge is implementing the adaption layer, integrating the system under test with the tool and model to make test cases executable. The final challenge is the dependency on a correct model representing the systems expected behaviour. These three challenges are also identified in the systematic literature review. Other challenges from the literature review include non-technical difficulties concerning training and the need to motivate staff and management.Using a limited case-study and a simple system three challenges, which are also identified in the literature review, throughout the MBT process are confirmed. MBT appears primarily as an academic subject with some examples of use in the industry.

Background. Speed is getting more and more critical in modern Software Engineering to be able to respond to users’ expectations of product development. One practice that takes a significant amount of time in the process of releasing software to the customers is testing. It is a clear trend that organizations are increasing the amount of test automation compared to manual testing. However, manual testing is still prominent in GUI-based testing due to challenges interacting with a GUI from test scripts. One of the most prominent challenges is synchronizing test script execution with the system under test. Objectives. This research aims to compare static and dynamic synchronization of GUI-based tests. This comparison will be conducted by replicating an existing Selenium test suite using static synchronization to Playwright, which will use dynamic synchronization and run the test suites against a web-based application. These test suites will then be used to compare the two types of synchronization from the perspective of test execution efficiency, test output correctness, and maintenance cost.Methods. The research methodology we chose is experiment. We have chosen Selenium to represent static synchronization and Playwright to represent dynamic synchronization. We used an existing test suite in Selenium that was translated into Playwright. There are a total of 81 tests in the test suite. The test suites were used to compare test scripts that use static and dynamic synchronization from the perspective of test execution efficiency, test output correctness, and maintenance cost.Results. The data collected from the experiments shows that execution efficiency for test scripts using dynamic synchronization is significantly faster than static synchronization. A mean difference between the test suites showed a decrease of 87%. For defect identification, ten defects where used and both test suites managed to identify all of them, resulting in no difference could be found. Test maintenance cost was found that test scripts using dynamic synchronization had a positive effect with an average of 60% less time spent on maintenance.Conclusions. Based on the result, we found that test scripts using dynamic synchronization improved execution efficiency and maintenance costs without sacrificing the test output correctness.

The goal of this thesis is to explore the possibility on if it is possible to automate regression testing for a SaaS application with a serverless approach. The thesis covers the fundamentals of the software development lifecycle, cloud concepts, different types of testing frameworks, and SaaS applications. The report researches various testing tools that can be used in accordance with Polestar’s needs. The testing framework must run the existing tests and deliver the results of the tests. The system must be able to coexist with the testing strategy that is in place today. The result is a testing framework that can run a number of selected tests on the SaaS application Salesforce. The system was deployed with serverless docker containers through Amazon Web Services. The report also covers what a future implementation can look like and potential improvements. / Målet med denna avhandling är att undersöka om det går att automatisera regression testning för en SaaS-applikation med hjälp av ett serverlöst tillvägagångsätt. Avhandlingen täcker grunderna inom molnkoncept, mjukvaruutveckling, olika typer av testramverk och SaaS-applikationer. Avhandlingen går även igenom gamla arbeten som har gjorts inom området. Rapporten behandlar val av verktyg med åtanke på vad företaget Polestar har för    behov. Testramverket ska kunna utföra automatiska regressionstester på SaaS-applikationen Salesforce. Resultatet är ett testramverk som kan köra ett antal utvalda tester på Salesforce. Systemet driftsattes med hjälp av serverlösa Docker containrar på Amazon Web Services. Avhandlingen täcker även alternativa verktyg som kan användas för testautomation och även potentiella förbättringsmöjligheter.