Mobile applications are growing in popularity and pose new problems in the area of software testing. In particular, mobile applications heavily depend upon user interactions and a dynamically changing environment of system events. In this thesis, we focus on user-driven events and use Q-learning, a reinforcement machine learning algorithm, to generate tests for Android applications under test (AUT). We implement a framework that automates the generation of GUI test cases by using our Q-learning approach and compare it to a uniform random (UR) implementation. A novel feature of our approach is that we generate user-driven event sequences through the GUI, without the source code or the model of the AUT. Hence, considerable amount of cost and time are saved by avoiding the need for model generation for generating the tests. Our results show that the systematic path exploration used by Q-learning results in higher average code coverage in comparison to the uniform random approach.

[pt] Uma estratégia comum de linha de produtos de software envolve sistemas da Web baseados em plug-ins que suportam a incorporação simples e rápida de comportamentos personalizados, sendo amplamente adotados para criar aplicativos baseados na web. A popularidade dos ecossistemas que suportam o desenvolvimento baseado em plug-ins (como o WordPress) é, em grande parte, devido ao número de opções de personalização disponíveis como plug-ins contribuídos pela comunidade. Entretanto, as vulnerabilidades relacionadas a plug-ins tendem a ser recorrentes, exploráveis e difíceis de serem detectadas e podem levar a graves conseqüências para o produto personalizado. Portanto, é necessário entender essas vulnerabilidades para permitir a prevenção de ameaças de segurança relevantes. Neste trabalho, realizamos um estudo exploratório para caracterizar vulnerabilidades causadas por plug-ins em sistemas baseados na web, examinando os boletins de vulnerabilidade do WordPress catalogados pelo National Vulnerability Database e os patches associados, mantidos pelo repositório de plugins do WordPress. Identificamos os principais tipos de vulnerabilidades, o seu impacto e o tamanho do patch para corrigir a vulnerabilidade. Identificamos, também, os tópicos mais comuns relacionados à segurança discutidos entre os desenvolvedores do WordPress. Observamos que, embora as vulnerabilidades possam ter consequências graves e permanecerem despercebidas por muito tempo, elas geralmente podem ser atenuadas com pequenas alterações no código-fonte. A caracterização ajuda a fornecer uma compreensão de como tais vulnerabilidades se manifestam na prática e contribui com as novas gerações de ferramentas de teste de vulnerabilidades capazes de antecipar sua possível ocorrência. Esta pesquisa propõe uma ferramenta de suporte para mitigar a ocorrência de vulnerabilidades em sistemas baseados em plugins web, facilitando a descoberta e antecipação da possível ocorrência de vulnerabilidades. / [en] A common software product line strategy involves plug-in-based web systems that support the simple and rapid incorporation of custom behaviors and are widely adopted for building web-based applications. The popularity of ecosystems that support plug-in-based development (such as WordPress) is largely due to the number of customization options available as community-contributed plugins. However, plug-in related vulnerabilities tend to be recurring, exploitable and difficult to detect and can lead to serious consequences for the custom product. Therefore, these vulnerabilities must be understood to enable the prevention of relevant security threats. In this paper, we conduct an exploratory study to characterize plug-in vulnerabilities in web-based systems by examining the WordPress vulnerability bulletins cataloged by the National Vulnerability Database and the associated patches maintained by the WordPress plugin repository. We identify the main types of vulnerabilities, their impact, and the size of the patch to address the vulnerability. We have also identified the most common securityrelated topics discussed among WordPress developers. We note that while vulnerabilities can have serious consequences and remain unnoticed for a long time, they can often be mitigated with minor changes to source code. Characterization helps provide an understanding of how such vulnerabilities manifest themselves in practice and contributes to new generations of vulnerability testing tools that can anticipate their potential occurrence. This research proposes a support tool to mitigate the occurrence of vulnerabilities in web plugin based systems, facilitating the discovery and anticipation of the possible occurrence of vulnerabilities.

<p>Cyber-Physical Systems (CPS) are integrated systems that consist of the computational and physical components with network communication to support operation in the physical world. My PhD dissertation focuses on the security and reliability of autonomous cyber-physical systems, such as self-driving cars, drones, and underwater robots, that are safety-critical systems based on the seamless integration of cyber and physical components. Autonomous CPS are becoming an integral part of our life. The market for autonomous driving systems is expected to be more than $65 billion by 2026. The security of such CPS is hence critical. Beyond traditional cyber-only computing systems, these complex and integrated CPS have unique characteristics. From the security perspective, they open unique research opportunities since they introduce additional attack vectors and post new challenges that existing cyber-oriented approaches cannot address well. <em>The goal of my research is to build secure and reliable autonomous CPS by bridging the gap between the cyber and physical domains.</em> To this end, my work focuses on fundamental research questions associated with cyber-physical attack and defense, vulnerability discovery and elimination, and post-attack investigation. My approach to solving the problems involves various techniques and interdis- ciplinary knowledge, including program analysis, search-based software engineering, control theory, robotics, and AI/machine learning.</p>

I takt med ökad användning av olika slags informationssystem växer även behovet av att testa mjukvaran, vilket innebär att allt fler vill komma igång med testautomation. Denna studie syftar till att beskriva hur man går tillväga för att ta första steget mot testautomationen med hjälp av Robot Framework, samt betona vikten av kravspecifikationer. Studien är en Design Science Research (designbaserad forskning) som löser ett verkligt problem på ett verkligt företag som i dagsläget anställer en hel avdelning av manuella testare och resulterar i en artefakt vilkens fullständig arkitektur redovisas i denna rapport. Problemet går ut på att automatisera ett testflöde, det vill säga en samling av tester som utförs manuellt. Testflödet behandlar den mest komplexa komponenten i hela systemet och tar en hel arbetsdag för manuella testare. Efter utförlig analys och automation förkortas testningstiden till tjugo minuter. Slutprodukten är en enkel applikation som guidar användaren igenom testet där användaren får välja vilka tester ska exekveras, samt sparar exekveringsresultat bestående av loggfiler och (om så önskats) en skärminspelning i en zip-fil. / As the usage of various information systems increases, so does the need to test software, meaning that more companies are interested in getting started with test automation. This study aims to describe how to proceed in order to take the first step towards test automation using Robot Framework, as well as to emphasize the importance of requirement specifications. The study is a Design Science Research, which is solving a real-life problem in a real company that currently employs a whole department of manual testers and is leading to the development of an artifact. The complete architecture of the artifact is disclosed in this report. The problem involves automating a test flow, that is, a collection of tests that are executed manually. The test is carried out on the most complex component in the entire system and takes a whole workday for manual testers. After thorough analysis and automation, the testing time is shortened to twenty minutes. The end product is a simple application that guides the user through the test where the user gets to choose which tests shall be executed, and saves the execution results consisting of log files and (if so wished) a screen recording in a zip file.

It would be difficult to imagine the world we live in without the World Wide Web. We depend on it for communication, entertainment, transfer of capital, access to essential services, and many other things. Even though it feels like the Web is everywhere, its usage is still growing, and so is its importance. However, a significant portion of the world’s population is made up of people with disabilities, and if the Web and its content is not made accessible to them, they cannot participate in this integral part of modern society. Making sure that the resources we access through the Web are accessible to people with disabilities is a difficult task. Those who create applications for the internet need to test them to identify accessibility issues. Today, much of the content on the Web is divided into units called components. It would be advantageous if there was a method for automated accessibility testing of these components. The problem is that no such method exists. Components—in this context—refer to the parts that make up a webpage. When you are, for example, viewing a news article online, the heading could be a component, as could the comments section, and so on. A majority of the world’s webpages are constructed using this architecture, where a set of reusable components with different functionality make up the page you are visiting. The purpose of this thesis is to create a method for automated accessibility testing of these web application components. The goal is that the created method should be useful for web developers and testers in their work to create a Web that is more accessible for people with disabilities, and therefore contribute in some way to a more accessible society. The chosen research methodology was qualitative and exploratory, and followed the design science research-paradigm. The methodology consisted of four distinct phases, a literature study phase, a preliminary design phase, an evaluation of the preliminary design phase, and an improved design phase. The literature study phase laid the groundwork for creating a method proposal in the preliminary design stage. This method was then evaluated in the evaluation stage. This evaluation consisted of a partial implementation of the framework, together with interviews with respondents with relevant experience. Using this evaluation, an improved method was created in the improved design phase. The result of this thesis is the Automated Accessibility Testing of Web Application Components Method (AAT-WAC). The evaluations that were conducted proved that AAT-WAC method met all of the stipulated evaluation criteria, and that the method was useful when implemented in a real-world industrial context. The literature study proved that no other methods similar to the AAT-WAC method existed. / Det skulle vara svårt att föreställa sig världen vi lever i utan Webben. Vi använder den för kommunikation, underhållning, överföring av kapital, tillgång till livsviktiga tjänster och mycket mer. Även om det kan kännas som att Webben redan är överallt så växer fortfarande dess användning och betydelse. En ansenlig del av världens befolkning är personer med funktionsnedsättningar, och om Webben och dess innehåll inte är tillgängligt för dem kan de inte deltaga i denna oumbärliga del av det moderna samhället. Att säkerställa att de resurser vi tillgodogör oss genom Webben är tillgängliga för personer med funktionsnedsättningar är en utmanande uppgift. De som skapar applikationer för Webben behöver testa dem för att upptäcka tillgänglighetsproblem. Idag är mycket av innehållet på Webben indelad i enheter som kallas för komponenter. Det skulle vara fördelaktigt om det fanns en metod för automatisk tillgänglighetstestning av dessa komponenter. Problemet är att ingen sådan metod existerar. Komponenter, i den här kontexten, syftar på de enheter som tillsammans utgör en webbsida. När du exempelvis besöker en nyhetsartikel på Webben så kan en komponent utgöra rubriken, en annan kommentarsektion, och så vidare. En majoritet av världens webbsidor är konstruerade enligt denna arkitektur, kallad en komponent-baserad arkitektur, där en uppsättning återanvändbara komponenter med olika funktionalitet utgör webbsidan du besöker. Syftet med denna avhandling är att skapa en metod för automatiserad tillgänglighetstestning av webbapplikationskomponenter. Målet är att öka kunskapen om tillgänglighetstestning, och genom detta skapa ett mer tillgängligt samhälle. Forskningsmetodologin var kvalitativ och följde en Design Science-paradigm. Forskningsmetodologin bestod av fyra distinkta faser, en litteraturstudie-fas, en preliminär designfas, en utvärderings av preliminär design-fas och en förbättrad design-fas. Litteraturstudien lade grunden för att skapa en preliminär metod i preliminär design-fasen. Denna preliminära metod utvärderades sedan genom att implementera delar av metoden på en verklig webapplikation, samt genom intervjuer med personer med relevant expertis inom webbutveckling. Dessa utvärderingar användes sedan för att skapa den förbättrade metoden i förbättrad design-fasen. Resultatet av denna avhandling är metoden Automated Accessibility Testing of Web Application Components (AAT-WAC-metoden). Uvärderingarna av AAT-WAC-metoden påvisade att metoden uppfyllde alla utvärderingskriterier som stipulerats, samt att metoden var användbar när den implementerades i en verklig, industriell kontext. Litteraturstudien påvisade att inga metoder som liknar AAT-WAC-metoden existerade.

Modern-day vehicles consist of numerous electronic computing devices with accompanying software. Since vehicles are generally classified as safety-critical systems, rigorous testing strategies have to be deployed to ensure correct operation of the embedded software. Testing in an active sense is understood to be the main testing method for software in general. The main characteristic of active testing is that test cases have complete control over the system under test, warranting sequential execution of test cases. To complement active testing a method called passive testing is being researched, where the main benefit compared to active testing is that the evaluation of test cases becomes parallelizable. As passive testing opens up for concurrency, the issue of not recognizing faulty behavior arises. This could be the case when simultaneously testing functions which share an output, and the incorrect behavior of a function is being masked by the correct behavior of another function. To avoid vacuous test results, the evaluation of dependent test cases can be separated. Previous work describes the process of extracting dependencies between test cases from requirement specifications. However, this approach is not suitable when test cases are derived from various artifacts. In this thesis we present a syntax-based approach for dependency discovery between test cases. The approach was evaluated through a case study using integration test cases developed for active hardware-in-the-loop testing at Scania CV. Dependencies between 946 test cases have been extracted and the test cases were grouped into three categories, with 286 test cases being identified as independent. The approach shows some potential for evaluating test cases written for active testing, but the results indicate that using test cases written specifically for passive testing may yield better results. The approach also proved to be useful for finding bugs and deviations in scripts. Thus, the method has proven useful for discovering dependencies from syntax, but the results are strongly dependent on the complexity and commonality of the scripts. / Moderna fordon innehåller åtskilliga elektroniska kontrollenheter med medföljande programvara. Eftersom fordon ofta anses vara säkerhetskritiska system krävs robusta teststrategier av den inbyggda mjukvaran för att säkerställa korrekt drift. Aktiv testning är för nuvarande den vanligaste metoden för att testa mjukvara. Den huvudsakliga egenskapen för aktiv testning är att testfall har total kontroll över systemet som är satt under test, vilket medför sekventiell utvärdering av testfall. För att komplettera aktiv testning undersöks en metod som kallas passiv testning, där den största fördelen gentemot aktiv testning är att utvärderingen av testfall kan ske parallellt. Att utvärdera testfall samtidigt medför dock att felaktigt beteende i mjukvaran potentiellt kan maskeras. Detta kan ske då flera funktioner delar utsignaler, och en funktions felaktiga beteende maskeras av en annan funktions korrekta beteende. För att undvika tomma sanningar som testresultat, kan utvärderingen av testfall som har ett beroende mellan dem separeras. Föregående arbeten inom området beskriver processen för att extrahera beroenden mellan testfall från kravspecifikationer. Denna metod är dock inte lämplig när testfall är baserade på flera olika artefakter. I detta examensarbete presenterar vi en syntaxbaserad metod för att hitta beroenden mellan testfall. Metoden utvärderades genom en fallstudie på Scania CV, där aktiva integrationstestfall för hardware-in-the-loop användes för analys. Beroenden mellan 946 testfall kunde extraheras och testfallen grupperades i tre olika kategorier, där 286 klassificerades som oberoende. Metoden visar potential för att analysera testfall skrivna för aktiv testning, men resultaten indikerar att testfall skrivna specifikt för passiv testning kan generera ett bättre resultat. Ett oväntat men positivt utfall är att metoden i viss grad kan användas för att hitta buggar och avvikelser i skript. Således har metoden visats användbar för att hitta beroenden från syntax, men resultatet beror starkt på skriptens komplexitet och gemensamhet.

Model-Based Testing has shown tangible benefits in previous research exploring widespread domains. Clearing systems in financial markets however are large software systems in which these approaches rarely have been explored. This paper in particular explores the Model-Based Testing approach using Modbat, a specialized tool based on extended finite-state machines, as a complementary testing approach on software clearing systems at Nasdaq. In particular, the approach is applied to functionalities of the system to test both its ability to find bugs undiscovered by traditional methods, as well as to measure its effectiveness and efficiency in creating reliable coverage in short amounts of time. In this study, we show that the model-based Modbat approach can achieve high code coverage with relatively little effort. 77 % of the production coverage was achieved with very few Modbat test runs. Additionally, only ∼ 9-14 % of the same lines of code, and a fraction of the time were required to achieve these results. From this, we conclude that Model-Based Testing approaches such as Modbat allow for significantly more efficient testing in terms of coverage over time compared to manual approaches. / Modellbaserad testning har visat på konkreta fördelar i tidigare forskning som utforskat omfattande områden. Clearingsystem inom finansmarknader är dock stora programvarusystem där dessa metoder sällan har utforskats. Denna arbete utforskar specifikt modellbaserad testning med användning av Modbat, ett specialiserat verktyg baserat på utökade ändliga automater, som ett kompletterande testningsmetodik för programvarurensningssystem på Nasdaq. Särskilt tillämpas metoden på specifika funktioner i systemet för att testa både dess förmåga att hitta buggar som inte upptäckts av traditionella metoder och att mäta dess effektivitet att skapa pålitlig täckning av kod på kort tid. I denna studie visar jag att den modellbaserade Modbat-metoden kan uppnå hög kodtäckning med relativt lite ansträngning. 77 % av produktionskoden täcktes med mycket få Modbat-testkörningar. Dessutom krävdes endast cirka 9-14 % av samma LOC, och en bråkdel av tiden för att uppnå dessa resultat.

Advancements within Large Language Models(LLMs) have opened a world of opportunities within the software development domain. This thesis, through an controlled experiment, aims to investigate how LLMs can be utilized within software testing, more specifically unit testing. The controlled experiment was performed using a Python script interfacing with the gpt-3.5-turbo model, to automatically generate unit tests for React Native components written in TypeScript. The pipeline described, performs the calls to the OpenAI Application Programming Interface(API) iterative. To evaluate and retrieve the metric code coverage, the unit tests were executed with Jest. Additionally, manual execution of failing tests, both compilable and non-compilable tests were executed and the different kind of errors with their frequency were documented. The experiment shows that LLMs can be used to generate comprehensive and accurate unit tests, with high potential of future improvements. While the amount of generated tests that compiled were low, their nature was often good, failing because of easy correctable syntax errors, faulty imports or missing dependencies. The errors found, were at large part due to project configurations while others would probably be less frequent through more extensive prompt-engineering or by the use of an newer model. The experiment also shows that the temperature affected the outcome and that the type of errors were different between compiling and non-compiling tests. A lower temperature parameter to the OpenAI API generally achieved better results, whilst a higher temperature showed greater coverage at compiled failing tests. This thesis also shows that future opportunities and improvements are widely available. Through better project optimization, newer models and better prompting, a better result is to be expected. The script could with further development be turned into a working product, making software testing faster and more efficient, saving both time and money while simultaneously improving the test case quality.

This thesis examines integration testing of object-oriented software. The process of integrating and testing procedural programs is reviewed as foundation for testing object-oriented software. The complexity of object-oriented software is examined. The relationship of integration testing and the software development life cycle is presented. Scenarios are discussed which account for the introduction of defects into the software. The Unified Modeling Language (UML) is chosen for representing pre-implementation and post-implementation models of the software. A demonstration of the technique of using post-implementation models representing the logical and physical views as an aid in integration and system testing of the software is presented. The use of UML diagrams developed from the software is suggested as a technique for integration testing of object-oriented software. The need for automating the data collection and model building is recognized. The technique is integrated into the Revised Spiral Model for Object-Oriented Software Development developed by du Plessis and van der Walt. / Computing / D.Phil. (Computer Science)

This thesis examines integration testing of object-oriented software. The process of integrating and testing procedural programs is reviewed as foundation for testing object-oriented software. The complexity of object-oriented software is examined. The relationship of integration testing and the software development life cycle is presented. Scenarios are discussed which account for the introduction of defects into the software. The Unified Modeling Language (UML) is chosen for representing pre-implementation and post-implementation models of the software. A demonstration of the technique of using post-implementation models representing the logical and physical views as an aid in integration and system testing of the software is presented. The use of UML diagrams developed from the software is suggested as a technique for integration testing of object-oriented software. The need for automating the data collection and model building is recognized. The technique is integrated into the Revised Spiral Model for Object-Oriented Software Development developed by du Plessis and van der Walt. / Computing / D.Phil. (Computer Science)