In this thesis we investigate the use of pairwise testing for PLC embedded software. We compare these automatically generated tests with tests created manually by industrial engineers. The tests were evaluated in terms of fault detection, code coverage and cost. In addition, we compared pairwise testing with randomly generated tests of the same size as pairwise tests. In order to automatically create test suites for PLC software a previously created tool called Combinatorial Test Tool (CTT) was extended to support pairwise testing using the IPOG algorithm. Once test suites were created using CTT they were executed on real industrial programs. The fault detection was measured using mutation analysis. The results of this thesis showed that manual tests achieved better fault detection (8% better mutation score in average) than tests generated using pairwise testing. Even if pairwise testing performed worse in terms of fault detection than manual testing, it achieved better fault detection in average than random tests of the same size. In addition, manual tests achieved in average 97.29% code coverage compared to 93.95% for pairwise testing, and 84.79% for random testing. By looking closely on all tests, manual testing performed equally good as pairwise in terms of achieved code coverage. Finally, the number of tests for manual testing was lower (12.98 tests in average) compared to pairwise and random testing (21.20 test in average). Interestingly enough, for the majority of the programs pairwise testing resulted in fewer tests than manual testing.

Testing every possible combination of the input parameter values is often impractical, inefficient or too expensive. One common alternative is pairwise testing where every pairwise combination of the parameter values is tested. Although pairwise testing significantly reduces the number of test cases, the challenge remains in analyzing the test outputs to discern the precise characteristics of parameters causing the failures. This thesis proposes a novel approach to output analysis by identifying “bad pairs”: pairs that always result in failed test cases. A framework implementing the proposed approach is presented together with three case studies. Results from the case studies suggest there are positive relationships among the numbers of failed test cases, faults, and independent bad pairs. Also, filtering of test cases seems to have a significant impact on the bad pairs identified. We believe the proposed approach can facilitate the debugging process in software testing.

Effective software performance testing is essential to the development and delivery of quality software products. Many software testing investigations have reported software performance testing improvements, but few have quantitatively validated measurable software testing performance improvements across an aggregate of studies. This study addressed that gap by conducting a meta-analysis to assess the relationship between applying Design of Experiments (DOE) techniques in the software testing process and the reported software performance testing improvements. Software performance testing theories and DOE techniques composed the theoretical framework for this study. Software testing studies (n = 96) were analyzed, where half had DOE techniques applied and the other half did not. Five research hypotheses were tested, where findings were measured in (a) the number of detected defects, (b) the rate of defect detection, (c) the phase in which the defect was detected, (d) the total number of hours it took to complete the testing, and (e) an overall hypothesis which included all measurements for all findings. The data were analyzed by first computing standard difference in means effect sizes, then through the Z test, the Q test, and the t test in statistical comparisons. Results of the meta-analysis showed that applying DOE techniques in the software testing process improved software performance testing (p < 05). These results have social implications for the software testing industry and software testing professionals, providing another empirically-validated testing methodology. Software organizations can use this methodology to differentiate their software testing process, to create more quality products, and to benefit the consumer and society in general.

Mjukvarutestning har, sedan introduktionen av datorer, varit föremål för forskning. Idag, när datorer och mikroprocessorer alltmer integreras i produkter som omger oss i vårt dagliga liv, så ökar vikten av effektiv och korrekt mjukvarutestning. Tidigare forskning visar att utforskande testning är en effektiv metod för att upptäcka programvarubuggar. Men, med tids- och resursaspekterna i åtanke finns det andra metoder som kan vara effektivare. En möjlig sådan metod är parvis testning. Litteraturen visar att även denna metod har bra potential för att kunna identifiera programvarubuggar. Det finns dock inte så mycket forskning om jämförelsen av dessa två metoder, vilket är anledningen till att denna studie genomfördes. Denna förklarande fallstudie utvärderar hur utforskande testning presterar jämfört med parvis testning. Aspekter som beaktades i utvärderingsprocessen var antalet upptäckta defekter, allvarlighetsgraden av de hittade defekterna och vilka typer av defekter som hittades. Data samlades in genom undersökningar, intervjuer, deltagande observationer och direkta observationer. Med all denna data som samlats in, drogs slutsatsen att svaret på denna rapports forskningsfrågor är tvetydiga. Det finns fördelar med båda teknikerna, och beroende på förutsättningarna för utforskande testning så kan parvis testning prestera likvärdigt. En sak märktes dock under studien som inte var en del av den ursprungliga omfattningen, och det var styrkan som utforskande testning har som ett läroverktyg. / Software testing has been a subject of research since the introduction of computers. Today, when computers and microprocessors are increasingly integrated into the products surrounding us in our daily lives, the importance of effective and accurate software testing increases. Previous research shows that exploratory testing is an effective method for detecting software bugs. Still, with the time and resource aspects considered, there are other potentially more time and resource-effective methods. One such possible method is the pairwise testing method. The literature also shows that this method is effective for finding software bugs. There is, however, not that much research about the comparison of these two methods, which is why this study was conducted. This explanatory case study evaluates how exploratory testing performs compared with pairwise testing. Aspects considered in the evaluation process were the number of detected defects, the severity distribution of the found defects, and what types of defects were found. The data was collected through surveys, interviews, participant observations, and direct observations. With the data collected, it was concluded that the answers to the research questions of this study are ambiguous. There are benefits with both of the techniques, and depending on the exploratory testing conditions, the pairwise technique can perform comparably as the exploratory testing. However, one thing noticed during the study that was not part of the original scope was the strength of exploratory testing as a learning tool. Lastly, some hypotheses were stated, supported by the collected data.

Embedded systems hold immense potential, but their integration into advanced devices comes with significant costs. Malfunctions in these systems can result inequipment failures, posing serious risks and potential accidents. To ensure theirproper functionality, embedded system components undergo rigorous testing phases,which can be time-consuming, especially for components with numerous connections. Therefore, it is crucial to reduce test time while maintaining high-qualitytesting to detect and address failures early in the development cycle, resulting in improved and safer products. This report delves into various techniques and algorithms aimed at expediting testingprocesses, such as machine learning, risk analysis, test parallelization, and combinatorial testing. It examines the practicality of mathematical models and automatedapproaches in real-world companies through experimentation and implementation.In essence, the report tackles the challenges involved in testing embedded systems,explores different approaches to reduce test time, and presents a suitable model formaintaining test quality. The ultimate goal is to present and implement a methodthat effectively reduces test time while upholding an acceptable level of test quality.The obtained results provide valuable insights for future test groups and researchersseeking to optimize their testing processes and deliver safer products