Background. Unstable and unpredictable tests, herein referred to as flaky tests, pose a serious challenge to systems in the production environment. If a device is not tested thoroughly, it will be sent back for retesting from the production centers, which is an expensive affair. Removing flaky tests involves detecting the flaky tests, finding the causes of flakiness and finally the elimination of flakiness. The existing literature provides information on causes and elimination techniques of flakiness for software systems. All of these are studied thoroughly, and support is taken from interviews to understand they are applicable in the context of embedded systems. Objectives. The primary objective is to identify causes of flakiness in a device under test and also techniques for eliminating flakiness. Methods. In this paper, we applied a literature review to find the current state-of-art of flakiness. A case study is selected to address the objectives of the study. Interviews and observations carried out to collect data. Data analysis performed using a directed content analysis method. Results. Observations resulted in eliminating 4 causes of flakiness in embedded systems. Interview results in finding 4 elimination techniques which were not found in the literature. Conclusions. Causes and Elimination techniques for the domain of embedded systems are identified. Knowledge translation between the domains was carried out effectively.

An important step in developing software is to test the system thoroughly. Testing software requires a generation of test cases that can reach large numbers and is important to be performed in the correct order. Certain information is critical to know to schedule the test cases incorrectly order and isn’t always available. This leads to a lot of required manual work and valuable resources to get correct. By instead analyzing their test specification it could be possible to detect the functional dependencies between test cases. This study presents a natural language processing (NLP) based approach and performs cluster analysis on a set of test cases to evaluate the correlation between test case dependencies and their semantic similarities. After an initial feature selection, the test cases’ similarities are calculated through the Cosine distance function. The result of the similarity calculation is then clustered using the HDBSCAN clustering algorithm. The clusters would represent test cases’ relations where test cases with close similarities are put in the same cluster as they were expected to share dependencies. The clusters are then validated with a Ground Truth containing the correct dependencies. The result is an F-Score of 0.7741. The approach in this study is used on an industrial testing project at Bombardier Transportation in Sweden.

[pt] A confiabilidade absoluta do software é considerada inatingível, pois mesmo quando confeccionado seguindo regras rígidas de qualidade, o software não está livre da ocorrência de falhas durante a sua vida útil. O nível de confiabilidade do software está relacionado, entre outros, à quantidade de defeitos remanescentes que serão exercitados durante seu uso. Contendo menos defeitos remanescentes, espera-se que o software falhe menos frequentemente, embora muitos desses defeitos sejam exercitados nenhuma vez durante a vida útil do software. Mas desenvolvedores, além de redigir programas, utilizam cada vez mais bibliotecas e serviços remotos que muitas vezes possuem qualidade duvidosa. Na tentativa de tornar o software capaz de observar erros em tempo de execução, surge a hipótese que o uso dos Métodos Formais Leves, por meio do emprego sistemático de assertivas executáveis, pode ser eficaz e economicamente viável para assegurar a confiabilidade do software, tanto em tempo de teste como em tempo de uso. O objetivo principal desta pesquisa é avaliar a eficácia de assertivas executáveis para prevenção e observação de falhas em tempo de execução. As avaliações da eficácia foram feitas por intermédio de uma análise quantitativa utilizando experimentos. Estes, utilizam, implementações de estruturas de dados instrumentadas com assertivas executáveis, submetidas a testes baseados em mutações. Os resultados mostraram que todos os mutantes não equivalentes foram identificados pelas assertivas, embora os testes não foram capazes disso. Também é apresentada uma estimativa do custo computacional relativo ao uso de assertivas executáveis. Com base na infraestrutura criada para realização dos experimentos é proposta uma política de instrumentação de programas utilizando assertivas executáveis a serem mantidas ativas tanto durante os testes como durante o uso produtivo. / [en] Absolute reliability of software is considered unattainable, because even when it is build following strict quality rules, software is not free of failure occurrences during its lifetime. Software s reliability level is related, among others, to the amount of remaining defects that will be exercised during its use. If software contains less remaining defects, it is expected that failures will occur less often, although many of these defects will never be exercised during its useful life. However, libraries and remote services of dubious quality are frequently used. In an attempt to enable software to check mistakes at runtime, hypothetically Lightweight Formal Methods, by means of executable assertions, can be effective and economically viable to ensure software s reliability both at test time as well as at run-time. The main objective of this research is to evaluate the effectiveness of executable assertions for the prevention and observation of run-time failures. Effectiveness was evaluated by means of experiments. We instrumented data structures with executable assertions, and subjected them to tests based on mutations. The results have shown that all non-equivalent mutants were detected by assertions, although several of them were not detected by tests using non-instrumented versions of the programs. Furthermore, estimates of the computational cost for the use of executable assertions are presented. Based on the infrastructure created for the experiments we propose an instrumentation policy using executable assertions to be used for testing and to safeguard run-time.

This thesis concerns the design, implementation and evaluation of a specification based testing architecture for reactive systems using the paradigm of learning-based testing. As part of this work we have designed, verified and implemented new incremental learning algorithms for DFA and Kripke structures.These have been integrated with the NuSMV model checker to give a new learning-based testing architecture. We have evaluated our architecture on case studies and shown that the method is effective. / QC 20110822

It is of utmost importance for software developers to develop a product that is functional for the end users. One way to ensure this is to have continuous unit tests forthe product. However, software testing is deemed as an unimportant part of the iteration process by many developers. The extent to which an open-source system hasbeen tested may show the current testing practices or lack thereof. The results couldhelp future developers and product owners to improve the quality of their softwareand help identify bugs early on in the iteration process. Previous studies have shownthat there is a trend amongst developers to avoid testing their products and this research helps to show if that is the case or not. We have created a tool that extractsthe method names, class names, method bodies, test classes and test methods froma project in order to analyse and show how thoroughly they are being tested. Theresults of this project have agreed with the previous studies that there is a tendencyamong developers to avoid testing which can lead to a lot of problems.

Testing is an important activity in software development used to ensure the quality of a product. In industrial practice when developing control software, such as Programmable Logic Controllers (PLCs), software testing is, in some cases, neglected during the development process. There is little support for automated testing on PLC software and the creation of test cases is hence mainly a manual activity that can be both costly and time-consuming. In this thesis, we propose a solution for an integrated tool chain for automating the testing process for IEC 61131-3 PLC software in the integrated development environment (IDE) of CODESYS (Controller Development System), using a combinatorial testing tool (i.e., SEAFOX) for test case generation. Further, we measure the applicability and usefulness of the integrated tool chain in terms of the decision coverage (i.e, Observable Decision Coverage) achieved by the generated test cases. For this purpose, several available tools for testing in CODESYS IDE have been examined as well as solutions on how to integrate the chosen tool (i.e., CfUnit) with SEAFOX. After a solution for the tool chain was implemented and integrated, an experiment was conducted to measure decision coverage for the generated test cases on nine industrial programs using Pairwise, Base choice and Random combinatorial testing techniques. The result of our thesis is a fully integrated tool chain consisting of CODESYS, CfUnit, SEAFOX, and a python script used to combine these, and where the SEAFOX tool was further extended to support additional standard data types. This tool chain can be used to create test cases, generate new input values for these using SEAFOX, automate the additional test case creation and integration with CfUnit in CODESYS IDE, as well as for test execution. The results of the experiment show that test cases generated by this tool chain achieved on average 90% decision coverage or higher regardless of the combinatorial technique used, with the overall average level being 94%. Interestingly enough, Random testing scored higher than both Pairwise and Base choice testing. We present our integrated tool chain as a contribution to the automation of test creation and execution for industrial control software in CODESYS IDE. However, we identified several limitations with testing CODESYS timer blocks and when executing larger test suites that can make it problematic to fully test PLC programs in a real-time simulation environment. These challenges should be investigated further by both researchers and practitioners.

Context: The core of software testing is test artifacts, i.e., test cases, test suites, test scripts, test code, test specifications, and natural language tests. Hence, the quality of test artifacts can negatively or positively impact the reliability of the software testing process. Several empirical studies and secondary studies have investigated the test artifact quality. Nevertheless, little is known about how practitioners by themselves perceive test artifact quality, and the evidence on test artifact quality in the literature has not been synthesized in one place. Objective: This thesis aims to identify and synthesize the knowledge in test artifact quality from both academia and industry. Hence, our objectives are: (1) To understand practitioners’ perspectives on test artifact quality, (2) To investigate how test artifact quality has been characterized in the literature, (3) To increase the reliability of the research method for conducting systematic literature reviews (SLR) in software engineering. Method: In this thesis, we conducted an interview-based exploratory study and a tertiary study to achieve the first two objectives. We used the tertiary study as a case and referred to related observations from other researchers to achieve the last objective. Results: We provided two quality models based on the findings of the interview-based and tertiary studies. The two models were synthesized and combined to provide a broader view of test artifact quality. Also, the context information that can be used to characterize the environment in which test artifact quality is investigated was aggregated based on these studies’ findings. Based on our experience in constructing and validating automated search results using a quality gold standard (QGS) in the tertiary study, we provided recommendations for the QGS construction and proposed an extension to the current search validation approach. Conclusions: The context information and the combined quality model provide a comprehensive view of test artifact quality. Researchers can use the quality model to develop guidelines, templates for designing new test artifacts, or assessment tools for evaluating existing test artifacts. The model also can serve as a guideline for practitioners to search for test-artifact quality information, i.e., definitions for the quality attributes and measurements. For future work, we aim at investigating how to improve relevant test artifact quality attributes that are challenging to deal with.

Background. Testing of a software system is a critical part of the software development process, as the need for efficient and high-quality software systems is of great importance. By using so-called test environments, the testing process can be automated and more effective, which in turn contributes to lower time and development costs. Objectives. This thesis aims to investigate the degree of usage of a test environment by introducing a model that measures the degree of usage for a test environment group. This thesis also aims to introduce suggestions for new measuring points to monitor the degree of usage for test environments. This thesis also presents suggestions for improvement measures that can be taken to increase the degree of usage for a test environment. Methods. The method that has been applied to carry out this thesis is a case study. The study began with a literature review followed by observations and semistructured interviews to identify factors that affect the measurement of the degree of usage. Also, archive research in form of log analysis has been carried out to analyze and retrieve data from machine logs. The suggested model has been evaluated through a static evaluation and the research has been carried out in collaboration with Försäkringskassan IT. Results. The results of this thesis are presented through a model that which is based on Goal Question Metric (GQM) approach. The model measures the degree of usage in a test environment. The model is based on reading data from machine logs, where the data reading befalls both during the night and day. The data reading can then be visualized by two (2) curves that present both an average usage (approximately 30 days) and a daily usage. These curves help to create an overall picture of the usage of a test environment, but also when during a day mostly usage happens. The visualization can then present through predefined limits on what is considered as good usage (the limits are different for different types of tests). To increase the measurement of the degree of usage, the model can be supplemented with measurement data from, among other things, the number of logins in the test environment, incoming and outgoing network traffic. Based on the analysis, measures such as sharing integration testing environments, and prioritize teams/projects that are working towards the next release can be taken to increase the degree of usage of a test environment. Conclusions. The degree of usage of a test environment can be measured by reading data from machine logs. In combination with measurement data from CPU, memory, and network traffic, the usage of a test environment can be measured and monitored, which in turn contributes to a higher degree of usage and efficiency around the testing process. / Bakgrund. Testning av ett mjukvarusystem utgör en kritisk del i utvecklingsprocessen, då behovet av effektiva och högkvalitativa mjukvarusystem är av stor betydelse.Genom att använda så kallade testmiljöer kan testningsprocessen automatiseras ocheffektiviseras, vilket i sin tur bidrar till lägre tids- och utvecklingskostnader.  Syfte. Syftet med detta examensarbete är att undersöka och utreda nyttjandegraden i en testmiljögrupp genom att introducera en modell som mäter nyttjandegraden för denna testmiljögrupp. Arbetet syftar också till att introducera förslag på nya mätpunkter för att övervaka mätningsgraden. Detta examensarbete presenterar även förslag på förbättringsåtgärder som kan tas vid för att öka nyttjandegraden kring en testmiljögrupp. Metod. Den metod som har tillämpats för att genomföra detta examensarbete är en fallstudie. Undersökningen inleddes med en litteraturstudie följt av observationer och semistrukturerade intervjuer, för att identifiera faktorer som påverkar mätning av nyttjandegraden. Även arkivforskning i form av logganalys har genomförts för att analysera och hämta data ur maskinloggar. Den förslagna modellen har utvärderats genom en statisk utvärdering. Forskningen har genomförts i samarbete med Försäkringskassan IT. Resultat. Resultatet av detta examensarbete presenteras i en modell som baseras på Goal Question Metric (GQM) tillvägagångssätt. Modellen mäter nyttjandegrad i en testmiljö. Modellen bygger på att utläsa data ur maskinloggar, där inläsningen sker både under natt och dag. Inläsningen kan sedan visualiseras genom två (2) kurvor som presenterar en genomsnittlig användning (förslagsvis 30 dagar) respektive en dygnanvändning. Dessa kurvor skapar en helhetsbild över såväl nyttjandet av testmiljön som när testmiljön används mest under ett dygn. Visualiseringen kan sedan slås ut på fördefinierade gränser över vad som anses vara ett gott nyttjande av testmiljö (gränserna är olika för olika typer av tester). För att bredda mätningen av nyttjandegraden kan modellen kompletteras med mätdata från bland annat antalet inloggningar som sker i testmiljön samt ingående och utgående nätverkstrafik. Baserat på den genomförda analysen kan vissa åtgärder vidtas för att öka nyttjandegraden av testmiljöer. Exempel på dessa åtgärder är gemensamt utnyttjande av integrationstestmiljöer och prioritering av team och/eller projekt som arbetar mot närmast release. Slutsatser. Nyttjandegrad kring testmiljöer kan mätas genom att utläsa data ur maskinloggar. I kombination med mätdata från CPU, minne och nätverkstrafik kan användning av testmiljöer mätas och övervakas, vilket i sin tur bidrar till ökad nyttjandegrad och effektivitet kring testningsprocessen.

Many real-time systems are developed and maintained through the use of commercial software products, such as Matlab and MatrixX, that automatically generate source code based on graphical control systems models. Testing these real-time models and the real-time software generated from them presents special problems during maintenance not faced with other forms of software. Very importantly, many of the models and software systems have to be tested through the use of simulations. Huge input and output data sets, the need for testing over a long duration of time (weeks or months), and computationally intensive requirements are just a few of the difficulties. For testing during maintenance in such situations, this paper draws upon field experience to present a set of test types and a strategy for selecting test types used to create series of input values to serve as test cases. Also this paper presents strategies for applying these test types, using the assistance of a free, widely available testing tool that automates test case generation, executes the simulations, and supports the analysis of the test results.

Inbyggda system finns i nästan alla enheter som används i vårt dagliga liv, som exempelvis mobiltelefoner, kylskåp och bilar. En del enheter kan vara betydligt känsligare än andra, vilket innebär att en bugg som existerar i ett system kan orsaka skada, till och med förlust av människoliv, eller orsakar ingen skada alls. Mjukvarutestning och mjukvarufelsökning genomförs för att reducera buggar i ett system.Utbildningsprogrammet Datateknik och Mobil IT på Malmö universitet fokuserar inte på att undervisa mjukvarufelsökning med hjälp av felsökningsverktyg. Således presenterar denna forskning en felsökningslaboration skapat för studenter som går Datateknik och Mobil IT, som anses hjälpa studenterna att få kunskap i hur man använder felsökningsverktyget SAM4E Xplained Pro för att lokalisera buggar. Som ett resultat, utfördes felsökningslaborationen av fyra studenter varav 75 procent av buggarna hittades och åtgärdades. / Embedded systems are found in almost every device used in our daily lives, including cell phones, refrigerators, and cars. Some devices may be significantly more sensitive than others, meaning a bug appearing in a system could cause harm, even loss of human lives or cause no harm at all. To reduce bugs in a system, software testing and software debugging are performed.The Computer Science program at Malmö University does not focus on teaching software debugging using a debugger. Thus, this thesis presents a debugging lab created for Computer Science students, considered to help them gain knowledge in how to use the debugger SAM4E Xplained Pro to locate bugs. As a result, four students performed the debugging lab of which 75 percent of the bugs were found and remedied.