Nowadays, deep neural networks based software have been widely applied in many areas including safety-critical areas such as traffic control, medical diagnosis and malware detection, etc. However, the software engineering techniques, which are supposed to guarantee the functionality, safety as well as fairness, are not well studied. For example, some serious crashes of DNN based autonomous cars have been reported. These crashes could have been avoided if these DNN based software were well tested. Traditional software testing, debugging or repairing techniques do not work well on DNN based software because there is no control flow, data flow or AST(Abstract Syntax Tree) in deep neural networks. Proposing software engineering techniques targeted on DNN based software are imperative. In this thesis, we first introduced the development of SE(Software Engineering) for AI(Artificial Intelligence) area and how our works have influenced the advancement of this new area. Then we summarized related works and some important concepts in SE for AI area. Finally, we discussed four important works of ours. Our first project DeepTest is one of the first few papers proposing systematic software testing techniques for DNN based software. We proposed neuron coverage guided image synthesis techniques for DNN based autonomous cars and leveraged domain specific metamorphic relation to generate oracle for new generated test cases to automatically test DNN based software. We applied DeepTest to testing three top performing self-driving car models in Udacity self-driving car challenge and our tool has identified thousands of erroneous behaviors that may lead to potential fatal crash. In DeepTest project, we found that the natural variation such as spatial transformations or rain/fog effects have led to problematic corner cases for DNN based self-driving cars. In the follow-up project DeepRobust, we studied per-point robustness of deep neural network under natural variation. We found that for a DNN model, some specific weak points are more likely to cause erroneous outputs than others under natural variation. We proposed a white-box approach and a black-box approach to identify these weak data points. We implemented and evaluated our approaches on 9 DNN based image classifiers and 3 DNN based self-driving car models. Our approaches can successfully detect weak points with good precision and recall for both DNN based image classifiers and self-driving cars. Most of existing works in SE for AI area including our DeepTest and DeepRobust focus on instance-wise errors, which are single inputs that result in a DNN model's erroneous outputs. Different from instance-wise errors, group-level errors reflect a DNN model's weak performance on differentiating among certain classes or inconsistent performance across classes. This type of errors is very concerning since it has been found to be related to many real-world notorious errors without malicious attackers. In our third project DeepInspect, we first introduced the group-level errors for DNN based software and categorized them into confusion errors and bias errors based on real-world reports. Then we proposed neuron coverage based distance metric to detect group-level errors for DNN based software without requiring labels. We applied DeepInspect to testing 8 pretrained DNN models trained in 6 popular image classification datasets, including three adversarial trained models. We showed that DeepInspect can successfully detect group-level violations for both single-label and multi-label classification models with high precision. As a follow-up and more challenging research project, we proposed five WR(weighted regularization) techniques to repair group-level errors for DNN based software. These five different weighted regularization techniques function at different stages of retraining or inference of DNNs including input phase, layer phase, loss phase and output phase. We compared and evaluated these five different WR techniques in both single-label and multi-label classifications including five combinations of four DNN architectures on four datasets. We showed that WR can effectively fix confusion and bias errors and these methods all have their pros, cons and applicable scenario. All our four projects discussed in this thesis have solved important problems in ensuring the functionality, safety as well as fairness for DNN based software and had significant influence in the advancement of SE for AI area.

In order to generate substantial market impact, mobile applications must be able to run on multiple platforms. Hence, software engineers face a multitude of technologies and system versions resulting in static variability. Furthermore, due to the dependence on sensors and connectivity, mobile software has to adapt its behavior accordingly at runtime resulting in dynamic variability. However, software engineers need to assure quality of a mobile application even with this large amount of variability—in our approach by the use of model-based testing (i.e., the generation of test cases from models). Recent concepts of test metamodels cannot efficiently handle dynamic variability. To overcome this problem, we propose a process for creating black-box test models based on dynamic feature Petri nets, which allow the description of configuration-dependent behavior and reconfiguration. We use feature models to define variability in the system under test. Furthermore, we illustrate our approach by introducing an example translator application.

The main part in the testing of the software is in the generation of test cases suitable for software system testing. The quality of the test cases plays a major role in reducing the time of software system testing and subsequently reduces the cost. The test cases, in model de- sign stages, are used to detect the faults before implementing it. This early detection offers more flexibility to correct the faults in early stages rather than latter ones. The best of these tests, that covers both static and dynamic software system model specifications, is one of the chal- lenges in the software testing. The static and dynamic specifications could be represented efficiently by Unified Modelling Language (UML) class diagram and sequence diagram. The work in this thesis shows that High Level Petri Nets (HLPN) can represent both of them in one model. Using a proper model in the representation of the software specifications is essential to generate proper test cases. The research presented in this thesis introduces novel and automated test cases generation techniques that can be used within a software sys- tem design testing. Furthermore, this research introduces e cient au- tomated technique to generate a formal software system model (HLPN) from semi-formal models (UML diagrams). The work in this thesis con- sists of four stages: (1) generating test cases from class diagram and Object Constraint Language (OCL) that can be used for testing the software system static specifications (the structure) (2) combining class diagram, sequence diagram and OCL to generate test cases able to cover both static and dynamic specifications (3) generating HLPN automat- ically from single or multi sequence diagrams (4) generating test cases from HLPN. The test cases that are generated in this work covered the structural and behavioural of the software system model. In first two phases of this work, the class diagram and sequence diagram are decomposed to nodes (edges) which are linked by Classes Hierarchy Table (CHu) and Edges Relationships Table (ERT) as well. The linking process based on the classes and edges relationships. The relationships of the software system components have been controlled by consistency checking technique, and the detection of these relationships has been automated. The test cases were generated based on these interrelationships. These test cases have been reduced to a minimum number and the best test case has been selected in every stage. The degree of similarity between test cases is used to ignore the similar test cases in order to avoid the redundancy. The transformation from UML sequence diagram (s) to HLPN facilitates the simpli cation of software system model and introduces formal model rather than semi-formal one. After decomposing the sequence diagram to Combined Fragments, the proposed technique converts each Combined Fragment to the corresponding block in HLPN. These blocks are con- nected together in Combined Fragments Net (CFN) to construct the the HLPN model. The experimentations with the proposed techniques show the effectiveness of these techniques in covering most of the software system specifications.

Model-based testing (MBT) is a promising testing method with advantages like exhaustive exploration and high maintainability. However, one notable downside is that the generated tests usually contain much unnecessary noise. This noise can present itself as superfluous actions that bear no effect on test outcome — worsening comprehensibility and inflating test size. Generalpurpose minimization techniques like delta debugging have been successful in minimizing similar input before. The process involves removing elements that are redundant for satisfying given criteria, e.g., that a test still identifies a specific fault. In this thesis, we formulate the modmin algorithm which makes use of a hierarchical delta debugging approach to minimize sequences generated with Modbat — an open source MBT tool based on the extended finite-state machine (EFSM). One after the other, the algorithm attacks three common sub-structures found within the generated tests: model instances, loops, and transitions. To evaluate the work, we extended Modbat with modmin and applied it to tests generated from a set of ten models of varying complexity. The results show that modmin is very proficient at minimizing the tests generated from our model set and that it does so at a negligible cost. / Modellbaserad testning är en lovande teknik med fördelar som uttömande sökning och hög underhållbarhet. En nackdel är däremot att de genererade testfallen tenderar att innehålla onödig information. Ett testfall ska, med fördel, vara så kort och koncist som möjligt, och överflödiga instruktioner förvärrar både testbegriplighet och teststorlek. Minimeringsstrategier som delta debugging har med goda resultat används för att minimera liknande datastrukturer tidigare. Processen innebär vanligtvis att man plockar bort element som inte är nödvändiga för att särskilda kriterier ska vara uppfyllda. Exempelvis att ett test fortfarande identifierar samma fel som innan. I det här verket formulerar vi och implementerar modmin-algoritmen, en algoritm som bygger på hierarkisk delta debugging för att minimera testfall generade med det modellbaserade testningsverktyget Modbat. En efter en attackerar vår algoritm tre vanliga delstrukturer som vi har identifierat i Modbats testfall: modellinstanser, slingor, och individuella övergångar. Vi utvärderade arbetet genom att utöka Modbats öppna källkod med modmin och sen minimera testfall genererade från tio olika modeller av varierande komplexitet. Resultaten visar att modmin klarar av att minimera testfall generade från alla våra modeller och att det bara tillför en försumbar kostnad vad gäller systemresurser eller körtid.

This dissertation introduces a hybrid strategy for automated testing of Android applications that combines reinforcement learning and test suite prioritization. These approaches aim to improve the effectiveness of the testing process by employing reinforcement learning algorithms, namely Q-learning and SARSA (State-Action-Reward-State-Action), for automated test case generation. The studies provide compelling evidence that reinforcement learning techniques hold great potential in generating test cases that consistently achieve high code coverage; however, the generated test cases may not always be in the optimal order. In this study, novel test case prioritization methods are developed, leveraging pairwise event interactions coverage, application state coverage, and application activity coverage, so as to optimize the rates of code coverage specifically for SARSA-generated test cases. Additionally, test suite prioritization techniques are introduced based on UI element coverage, test case cost, and test case complexity to further enhance the ordering of SARSA-generated test cases. Empirical investigations demonstrate that applying the proposed test suite prioritization techniques to the test suites generated by the reinforcement learning algorithm SARSA improved the rates of code coverage over original orderings and random orderings of test cases.

Embedded systems hold immense potential, but their integration into advanced devices comes with significant costs. Malfunctions in these systems can result inequipment failures, posing serious risks and potential accidents. To ensure theirproper functionality, embedded system components undergo rigorous testing phases,which can be time-consuming, especially for components with numerous connections. Therefore, it is crucial to reduce test time while maintaining high-qualitytesting to detect and address failures early in the development cycle, resulting in improved and safer products. This report delves into various techniques and algorithms aimed at expediting testingprocesses, such as machine learning, risk analysis, test parallelization, and combinatorial testing. It examines the practicality of mathematical models and automatedapproaches in real-world companies through experimentation and implementation.In essence, the report tackles the challenges involved in testing embedded systems,explores different approaches to reduce test time, and presents a suitable model formaintaining test quality. The ultimate goal is to present and implement a methodthat effectively reduces test time while upholding an acceptable level of test quality.The obtained results provide valuable insights for future test groups and researchersseeking to optimize their testing processes and deliver safer products

Software testing is a crucial development technique to capture defects and slow code. When testing 3D graphics, it is hard to create automatic tests that detect errors or slow performance. Finding performance issues in game maps is a complex task that requires much manual work. Gaming companies such as EA DICE could benefit from automating the process of finding these performance issues in their game maps. This thesis tries to solve the problem by creating automatic tests where the camera is placed in a top-down perspective and flies over the in-game map, recording the time it takes to create render and client simulation frames for each map segment. The resulting trace is then visualised as a heatmap, where the mean frame creation times are rendered with pseudo colouring techniques to help pinpoint possible issues for the test engineers. The key findings of this thesis are that a heatmap visualisation of frame creation times saves much time for the developers trying to find these issues; it also lowers the amount of knowledge needed to find performance issues. This tool automates a process that formerly needed considerable manual work to get the same result. Now, artists with low coding experience can find performance issues without the technical knowledge of a Quality Assurance engineer. The thesis also highlights the drawbacks of a top-down perspective of camera trace since this is not how EA DICE games are usually rendered for the player in runtime. With this thesis as a base, other tests could be made with other ways of moving the camera and visualising the trace. / Mjukvarutestning är en viktig programvaruutvecklings teknik för att fånga felaktig eller långsam kod. Det är svårt att skapa automatiska tester för 3D grafik som hittar fel eller dålig prestanda i koden. Att hitta prestandaproblem i spelkartor är en komplex uppgift som kräver mycket manuellt arbete. Spelföretag såsom EA DICE skulle dra fördel av att automatisera processen att hitta dessa prestandaproblem i spelkartor. Denna uppsats försöker lösa detta genom att skapa automatiska tester där kameran placeras i ett uppifrån-och-ned-perspektiv och sedan flyger genom banan i spelet samtidigt som den samlar in data på hur lång tid det tar för renderings-bildrutor och klient-simulerings-bildrutor att skapas för varje ban-segment. Den resulterande datan är därefter visualiserade som ett färgdiagram, där medelvärdet på tiden för att skapa varje bildruta ritas upp med en psuedofärgningsteknik för att markera möjliga problemområden för testingenjörerna. Nyckelupptäckter för denna uppsats är att färgdiagramsvisualiseringen av bildruta-skapande-tider sparar mycket tid för utvecklare som försöker hitta prestandaproblem. Det minskar också kunskapströskeln som behövs för att lokalisera prestandaproblem. Detta verktyg automatiserar en process som tidigare krävde omfattande manuellt arbete för att få samma resultat. Numera kan game artists med låg koderfarenhet hitta dessa prestandaproblem utan den tekniska kunskapen hos en kvalitetskontroll-ingenjör. Den här uppsatsen visar också nackdelar med ett uppifrån-och-ned-perspektiv för kameran då det inte är så EA DICE spel normalt renderas för spelarna. Den här uppsatsen kan användas som utgångspunkt för andra som vill utveckla testverktyg och med fördel ta i beaktning de utvecklingspunkter denna uppsats belyser.

Quality assurance and software testing of software artifacts is as important as ever and this is especially so the case in web applications. The web applications of today are more complex and are used in more critical systems at a larger scale than ever before. However, testing of these applications is very challenging due to their dynamic nature. It is somewhat challenging to find clear and up-to-date guidelines on how to implement and evaluate regression software testing in small and medium-size enterprises (SME’s) developing web applications. The purpose of this thesis was to investigate the problem at hand and propose an approach to implementing software regression testing in web applications for SME’s. That is, recommending what to test, recommending what kind of software testing could be implemented and using what state of the art front end testing frameworks. An in-depth literature study was conducted to see what had been done in the past and present. Two rounds of semi structured in-depth interviews were conducted with software developers at the company where this thesis was conducted. The main purpose of the first interview was to find business goals from which to derive and subsequently create a testing suite in four different testing frameworks; Cypress, Jest, Playwright and Vitest. The purpose of the second interview was to evaluate and compare the aforementioned testing suites in order to propose an approach on software testing in web applications. In addition, code coverage and mutation scoring was also considered when evaluating the testing suites. The findings of this thesis is that a reasonable approach of introducing software testing into an SME which develops a web application, is to use business requirements for generating test cases and prioritizing end-to-end testing since the perceived benefits of the latter in this thesis far outweigh the benefits of the component testing suites although a combination of both would be the best of both worlds. Although this thesis was conducted on a web application written in React, the findings and recommendations can be applied to any front end framework such as Angular or Vue. / Kvalitetssäkring och testning av mjukvara är lika viktigt som alltid och detta är i synnerhet även fallet i webbapplikationer. Dagens webbapplikationer är mer komplexa och används i mer kritiska system på en större skala än någonsin tidigare. Dessvärre är det svårt att testa dessa applikationer eftersom att de är dynamiska. Det är svårt att hitta riktlinjer för hur man ska implementera och utvärdera regressionstester på små och medelstora företag (SMF) som utvecklar webbapplikationer. Syftet med denna uppsats var att undersöka problemet och föreslå en riktlinje för hur man kan implementera regressionstestning i SMF och i webbapplikationer. Detta innebär att föreslå vad man kan testa, vilken form av mjukvarutestning man kan implementera och med vilka moderna testningsramverk man kan göra detta med. En ingående litteraturstudie genomfördes för att ta reda på vad som hade gjorts tidigare inom området. Två rundor av semistrukturerade intervjuer genomfördes med mjukvaruutvecklarna på företaget där uppsatsen genomfördes. Syftet med den första intervjun var att hitta företagsmål som sedan agerade grund till testningssviter i fyra olika ramverk; Cypress, Jest, Playwright och Vitest. Syftet med den andra intervjun var att utvärdera och jämföra dessa testsviter för att rekommendera ett tillvägagångssätt för att implementera mjukvarutestning i webbapplikationer. Utöver intervjuerna så bidrog mutationspoäng och kodtäckning till rekommendationerna. Uppsatsen finner att ett rimligt sätt att implementera regressionstester i ett SMF och en webbapplikation är att generera testfall utifrån affärskrav och att prioritera testning på användarnivå eftersom att fördelarna från denna nivå av testning överväger fördelarna från komponenttestning. Allra helst bör man implementera en kombination av båda nivåerna. Fastän denna uppsats undersökte en webbapplikation i React så kan dessa upptäckter och rekommendationer även tillämpas på vilket frontendramverk som helst så som exempelvis Angular eller Vue.

With the increasing pace of development, the process of interpreting software test results data has become more challenging and time-consuming. While the test results provide valuable insights into the software product, the increasing complexity of software systems and the growing volume of test data pose challenges in effectively analyzing this data to ensure quality. To address these challenges, organizations are adopting various tools. Visualization dashboards are a common approach used to streamline the analysis process. By aggregating and visualizing test results data, these dashboards enable easier identification of patterns and trends, facilitating informed decision-making. This study proposes a management dashboard with visualizations of test results data as a decision support system. A case study was conducted involving eleven quality assurance experts with a number of various roles, including managers, directors, testers, and project managers. User interviews were conducted to evaluate the need for a dashboard and identify relevant test results data to visualize. The participants expressed the need for a dashboard, which would benefit both newcomers and experienced employees. A low-fidelity prototype of the dashboard was created and A/B testing was performed through a survey to prioritize features and choose the preferred version of the prototype. The results of the user interviews highlighted pass-rate, executed test cases, and failed test cases as the most important features. However, different professions showed interest in different test result metrics, leading to the creation of multiple views in the prototype to accommodate varying needs. A high-fidelity prototype was implemented based on feedback and underwent user testing, leading to iterative improvements. Despite the numerous advantages of a dashboard, integrating it into an organization can pose challenges due to variations in testing processes and guidelines across companies and teams. Hence, the dashboards require customization. The main contribution of this study is twofold. Firstly, it provides recommendations for relevant test result metrics and suitable visualizations to effectively communicate test results. Secondly, it offers insights into the visualization preferences of different professions within a quality assurance team that were missing in previous studies. / Med den ökande utvecklingstakten har processen att tolka testresultatdata för programvara blivit mer utmanande och tidskrävande. Även om testresultaten ger värdefulla insikter i mjukvaruprodukten, innebär den ökande komplexiteten hos mjukvarusystemen och den växande volymen testdata utmaningar när det gäller att effektivt analysera dessa data för att säkerställa kvalitet. För att möta dessa utmaningar använder organisationer olika verktyg. Visualiseringspaneler är ett vanligt tillvägagångssätt som används för att effektivisera analysprocessen. Genom att aggregera och visualisera testresultatdata möjliggör dessa instrumentpaneler enklare identifiering av mönster och trender, vilket underlättar välgrundat beslutsfattande. Den här studien föreslår en management-panel med visualiseringar av testresultatdata som ett beslutsstödssystem. En fallstudie genomfördes med elva experter inom kvalitetssäkring med olika roller, inklusive chefer, direktörer, testare och projektledare. Användarintervjuer genomfördes för att utvärdera behovet av en panel och identifiera relevanta testresultatdata att visualisera. Deltagarna uttryckte behovet av en visualiseringspanel, som skulle gynna både nyanställda och erfarna medarbetare. En prototyp av panelen med låg detaljnivå skapades och A/B-testning genomfördes genom en enkät för att prioritera funktioner och välja den föredragna versionen av prototypen. Resultaten av användarintervjuerna lyfte fram andel av godkända testresultat, exekverade testfall och misslyckade testfall som de viktigaste egenskaperna. Men olika yrkesgrupper visade intresse för olika testresultatmått, vilket ledde till skapandet av flera vyer i prototypen för att tillgodose olika behov. En prototyp med hög detaljnivå implementerades baserat på feedback och genomgick användartestning, vilket ledde till iterativa förbättringar. Trots de många fördelarna med en instrumentpanel kan det innebära utmaningar att integrera den i en organisation på grund av variationer i testprocesser och riktlinjer mellan företag och team. Därför kräver paneler anpassning. Det huvudsakliga bidraget från denna studie är dubbelt. För det första ger den rekommendationer för relevanta testresultatmått och lämpliga visualiseringar för att effektivt kommunicera testresultat. För det andra ger den insikter i visualiseringspreferenser för olika yrken inom ett kvalitetssäkringsteam vilket saknats i tidigare studier.

Software testing plays a fundamental role in software engineering as it ensures the quality of a software system. However, one of the major challenges of software testing is its costs since it is a time and resource-consuming process which according to academia and industry can take up to 50% of the total development cost. Today, one of the most common ways of generating testcases is through manual labor by analyzing specification documents to produce test scripts, which tends to be an expensive and error prone process. Therefore, optimizing software testing by automating the test case generation process can result in time and cost reductions and also lead to better quality of the end product. Currently, most of the state-of-the-art solutions for automatic test case generation require the usage of formal specifications. Such formal specifications are not always available during the testing process and if available, they require expert knowledge for writing and understanding them. One artifact that is often available in the testing domain is test case specifications written in natural language. In this thesis, an approach for generating integration test cases from natural language test case specifications is designed, applied and, evaluated. Machine learning and natural language processing techniques are used to implement the approach. The proposed approach is conducted and evaluated on an industrial testing project at Ericsson AB in Sweden. Additionally, the approach has been implemented as a tool with a graphical user interface for aiding testers in the process of test case generation. The approach involves performing natural language processing techniques for parsing and analyzing the test case specifications to generate feature vectors that are later mapped to label vectors containing existing C# test scripts filenames. The feature and label vectors are used as input and output, respectively, in a multi-label text classification process. The approach managed to produce test scripts for all test case specifications and obtained a best F1 score of 89% when using LinearSVC as the classifier and performing data augmentation on the training set. / Programvarutestning spelar en grundläggande roll i programvaruutveckling då den säkerställer kvaliteten på ett programvarusystem. En av de största utmaningarna med programvarutestning är dess kostnader eftersom den är en tids och resurskrävande process som enligt akademin och industrin kan ta upp till 50% av den totala utvecklingskostnaden. Ett av de vanligaste sätten att generera testfall idag är med manuellt arbete genom analys av testfallsspecifikationer, vilket tenderar att vara en dyr och felbenägen process. Därför kan optimering av programvarutestning genom automatisering av testfallsgenereringsprocessen resultera i tids- och kostnadsminimeringar och även leda till bättre kvalitet på slutprodukten. Nuförtiden kräver de flesta toppmoderna lösningarna för automatisk testfallsgenerering användning av formella specifikationer. Sådana specifikationer är inte alltid tillgängliga under testprocessen och om de är tillgängliga, så krävs det expertkunskap för att skriva och förstå dem. En artefakt som ofta finns i testdomänen är testfallspecifikationer skrivna på naturligt språk. I denna rapport utformas, tillämpas och utvärderas en metod för generering av integrationstestfall från testfallsspecifikationer skrivna på naturligt språk. Maskininlärnings- och naturlig språkbehandlingstekniker används för implementationen av metoden. Den föreslagna metoden genomförs och utvärderas vid ett industriellt testprojekt hos Ericsson AB i Sverige. Dessutom har metoden implementerats som ett verktyg med ett grafiskt användargränssnitt för att hjälpa testare i testfallsgenereringsprocessen. Metoden fungerar genom att utföra naturlig språkbehandlingstekniker på testfallsspecifikationer för att generera egenskapsvektorer som senare mappas till etikettsvektorer som innehåller befintliga C# testskriptfilnamn. Engenskaps och etikettsvektorerna används sedan som indata och utdata, respektive, för textklassificeringsprocessen. Metoden lyckades producera testskript för allatestfallsspecifikationer och fick en bästa F1 poäng på 89% när LinearSVC användes för klassificeringen och datautökning var utförd på träningsdatat.