The objective of this thesis is to analyse theconnection between test results and field claims of ECUs (electronic controlunits) at Scania in order to improve the acceptance criteria and evaluatesoftware testing strategies. The connection is examined through computation ofdifferent measures of dependencies such as the Pearson’s correlation, Spearman’srank correlation and Kendall’s tau. The correlations are computed from testresults in different ECU projects and considered in a predictive model based onlogistic regression. Numerical results indicate a weak connection between testresults and field claims. This is partly due to insufficient number of ECUprojects and the lack of traceability of field claims and test results. Themain conclusion confirms the present software testing strategy. Continuoussoftware release and testing results in a lower field claim and thus a betterproduct.

The focus of this study is on examining when employees in the IT industry experience that it is more favorable to use automated tests and manual tests, respectively. The purpose of this study is to investigate how different companies in practice, use, work with and think about the two different test methods. Four factors that influence the choice of test method have been developed as a workframe based on five articles, all of which discuss the requirements and criteria for the two different test methods. By conducting an interview study with semi-structured interviews, data has been retrieved from two different companies. The data has in turn been analyzed based on the four factors that have been developed, namely: the number of test cases/test runs, technical aspects, what functions that are to be tested and resources. Based on the analyzes, it has been clear that the opinions from the different respondents and from previous research often remain on the same track. Thus, the various criteria could be discussed and the motivation for when the respondents use each method could be outlined. However, it also becomes clear that in the end it is mainly resources, often the number of working hours and the monetary cost, that determines what practice of testing is to be used. / Denna studie riktar in sig på att undersöka när anställda inom IT-branschen upplever att det är mer gynnsamt att använda sig av automatiserade tester respektive manuella tester. Syftet med studien är att undersöka hur olika företag använder sig, arbetar med och ser på de två olika testmetoderna i praktiken. Fyra stycken faktorer som påverkar valet av testmetod har tagits fram som ramverk som är baserad på fem artiklar där samtliga diskuterar krav och kriterier för de två olika testmetoderna. Genom att utföra en intervjustudie med semi-strukturerade intervjuer har data samlats in från två olika företag. Den insamlade datan har i sin tur analyserats utifrån de olika faktorerna som tagits fram nämligen: Antalet testfall/testkörningar, tekniska aspekter, funktioner som ska testas samt resurser. Utifrån analysen har det varit tydligt att åsikterna från de olika respondenterna samt från tidigare forskning ofta är på samma spår. Därmed har de olika kriterierna kunnat diskuterats och motiveringarna till när respondenterna använder sig av respektive metod kunnat benas ut. Dock blir det också tydligt att i slutändan är det resurser, oftast antalet arbetstimmar och den monetära kostnaden, som styr vad som borde väljas.

Effective software performance testing is essential to the development and delivery of quality software products. Many software testing investigations have reported software performance testing improvements, but few have quantitatively validated measurable software testing performance improvements across an aggregate of studies. This study addressed that gap by conducting a meta-analysis to assess the relationship between applying Design of Experiments (DOE) techniques in the software testing process and the reported software performance testing improvements. Software performance testing theories and DOE techniques composed the theoretical framework for this study. Software testing studies (n = 96) were analyzed, where half had DOE techniques applied and the other half did not. Five research hypotheses were tested, where findings were measured in (a) the number of detected defects, (b) the rate of defect detection, (c) the phase in which the defect was detected, (d) the total number of hours it took to complete the testing, and (e) an overall hypothesis which included all measurements for all findings. The data were analyzed by first computing standard difference in means effect sizes, then through the Z test, the Q test, and the t test in statistical comparisons. Results of the meta-analysis showed that applying DOE techniques in the software testing process improved software performance testing (p < 05). These results have social implications for the software testing industry and software testing professionals, providing another empirically-validated testing methodology. Software organizations can use this methodology to differentiate their software testing process, to create more quality products, and to benefit the consumer and society in general.

No description available.

<p>Binary analysis, a cornerstone technique in cybersecurity, enables the examination of binary executables, irrespective of source code availability.</p> <p>It plays a critical role in understanding program behaviors, detecting software bugs, and mitigating potential vulnerabilities, specially in situations where the source code remains out of reach.</p> <p>However, aligning the efficacy of binary analysis with that of source-level analysis remains a significant challenge, primarily due to the uncertainty caused by the loss of semantic information during the compilation process.</p> <p><br></p> <p>This dissertation presents an innovative probabilistic approach, termed as <em>probabilistic binary analysis</em>, designed to combat the intrinsic uncertainty in binary analysis.</p> <p>It builds on the fundamental principles of program sampling and probabilistic inference, enhanced further by an iterative refinement architecture.</p> <p>The dissertation suggests that a thorough and practical method of sampling program behaviors can yield a substantial quantity of hints which could be instrumental in recovering lost information, despite the potential inclusion of some inaccuracies.</p> <p>Consequently, a probabilistic inference technique is applied to systematically incorporate and process the collected hints, suppressing the incorrect ones, thereby enabling the interpretation of high-level semantics.</p> <p>Furthermore, an iterative refinement mechanism is deployed to augment the efficiency of the probabilistic analysis in subsequent applications, facilitating the progressive enhancement of analysis outcomes through an automated or human-guided feedback loop.</p> <p><br></p> <p>This work offers an in-depth understanding of the challenges and solutions related to assessing low-level program representations and systematically handling the inherent uncertainty in binary analysis. </p> <p>It aims to contribute to the field by advancing the development of precise, reliable, and interpretable binary analysis solutions, thereby setting the groundwork for future exploration in this domain.</p>

Unit testing plays a crucial role in ensuring the quality and reliability of software systems. However, manual testing can often be a slow and time-consuming process. With current advancements in artificial intelligence (AI), new tools have emerged for automated unit testing to address this issue. But how do these new AI tools compare to conventional automated unit test generation tools? To answer this question, we compared two state-of-the-art conventional unit test tools (EVOSUITE and RANDOOP) with the sole commercially available AI-based unit test tool (DIFFBLUE COVER) for Java. We tested them on 10 sample classes from 3 real-life projects provided by the Defects4J dataset to evaluate their performance regarding code coverage, mutation score, and fault detection. The results showed that EVOSUITE achieved the highest code coverage, averaging 89%, while RANDOOP and DIFFBLUE COVER achieved similar results, averaging 63%. In terms of mutation score, DIFFBLUE COVER had the lowest average score of 40%, while EVOSUITE and RANDOOP scored 67% and 50%, respectively. For fault detection, EVOSUITE and RANDOOP detected a higher number of bugs (7 out of 10 and 5 out of 10, respectively) compared to DIFFBLUE COVER, which found only 4 out of 10. Although the AI-based tool was outperformed in all three criteria, it still shows promise by being able to achieve adequate results, in some cases even surpassing the conventional tools while generating a significantly smaller number of total assertions and more comprehensive tests. Nonetheless, the study acknowledges its limitations in terms of the restricted number of AI-based tools used and the small number of projects utilized from Defects4J.

<p>This thesis presents an in-depth investigation into the opportunities of optimizing the performance (throughput) of fuzzing on Java Virtual Machine (JVM)-based languages. The study identifies five main areas for potential optimization, each of which contributes to the performance bottlenecks in the existing state-of-the-art Java fuzzer, Jazzer.</p> <p><br></p> <p>Firstly, the use of coverage probes is recognized as costly due to the native method call, including call frame generation and destruction, while it only performs a simple byte increment. Secondly, the probes may become exhausted, which subsequently cease to generate signals for new interesting inputs, while the associated costs persist. Thirdly, the scanning of the coverage map is expensive, particularly for targets with a large loaded bytecode. Given that test inputs can only execute a portion of these, the probes for most bytecodes are scanned repeatedly without generating any signals, indicating a need for a more structured coverage map design to skip the code probes effectively. Lastly, exception handling in JVM is costly as it automatically fills in the stack trace whenever an exception object is created, even when most targets don't utilize this information. </p> <p><br></p> <p>The study then designs and implements optimization techniques for these opportunities. We believe we provide the optimal solution for the first opportunity, while better optimizations could be proposed for the second, third, and fourth. The collective improvement brought about by these implementations is on average 138% and up to 441% in throughput. This work, thus, offers valuable insights into enhancing the efficiency of fuzz testing in JVM languages and paves the way for further research in optimizing other areas of JVM-based-language fuzzing performance.</p>

The criticality of the applications modeled by the real-time software places stringent requirements on software quality before deploying into real use. Though automated test tools can be used to run a large number of tests efficiently, the functionality of any test tool is not complege without providing a means for analyzing the test results to determine potential problem sub-domains and sub-domains that need to be covered, and estimating the reliability of the modeled system. This thesis outlines a solution strategy and implementation of that strategy for deriving quantitative metrics from domain testing of real-time control software tested via simulation. The key portion of this thesis addresses the combinatorial problems involved with effective evaluation of test coverage and provides the developer with reliability metrics from testing of the software to gain confidence in the test phase of development. The two approaches for reliability analysis- time domain and input domain approaches are studied and a hybrid approach that combines the strengths of both these approaches is proposed. A Reliability analysis Test Tool (RATT) has been developed to implement the proposed strategies. The results show that the metrics are practically feasible to compute and can be applied to most real-time software.

Software testing, specifically in the medical device field, has become increasingly complex over the last decade. Technological enhancements to simulate clinical scenarios and advancements in communicating to medical devices have created the need for better testing strategies and methodologies. Typical medical device companies have depended on manual testing processes to fulfill Food and Drug Administration (FDA) submission requirements specifically Class III devices which are life supporting, life sustaining devices. At St. Jude Medical, software testing of Class III devices such as implantable cardioverter-defibrillators (ICDs), pacemakers, and pacing analyzers are given top priority to ensure the highest quality in each product. High emphasis is made on improving software testing for ease of use and for catching more software errors in each device. A significant stride in testing has automated the process and has provided software verification teams with the tools they need to successfully test and deliver high quality products. By creating software libraries which interact with communication to the other interfaces needed to test medical devices, test engineers can focus on fully testing device requirements and will not be concerned with how each test will interact with the device or any other testing tools. The main focus will be a specific St. Jude Medical device known as the Pacing System Analyzer (PSA). The PSA device will be used to demonstrate how verification engineers are able to benefit from software libraries and allow the testing process and test development to be fully automated. New technologies and standards will be created to simulate clinical scenarios and to communicate to new devices. The goal is to use software engineering principles to create standard test libraries which sustain these changes while still allowing testers to focus on finding issues for each device.

Mjukvarutestning har, sedan introduktionen av datorer, varit föremål för forskning. Idag, när datorer och mikroprocessorer alltmer integreras i produkter som omger oss i vårt dagliga liv, så ökar vikten av effektiv och korrekt mjukvarutestning. Tidigare forskning visar att utforskande testning är en effektiv metod för att upptäcka programvarubuggar. Men, med tids- och resursaspekterna i åtanke finns det andra metoder som kan vara effektivare. En möjlig sådan metod är parvis testning. Litteraturen visar att även denna metod har bra potential för att kunna identifiera programvarubuggar. Det finns dock inte så mycket forskning om jämförelsen av dessa två metoder, vilket är anledningen till att denna studie genomfördes. Denna förklarande fallstudie utvärderar hur utforskande testning presterar jämfört med parvis testning. Aspekter som beaktades i utvärderingsprocessen var antalet upptäckta defekter, allvarlighetsgraden av de hittade defekterna och vilka typer av defekter som hittades. Data samlades in genom undersökningar, intervjuer, deltagande observationer och direkta observationer. Med all denna data som samlats in, drogs slutsatsen att svaret på denna rapports forskningsfrågor är tvetydiga. Det finns fördelar med båda teknikerna, och beroende på förutsättningarna för utforskande testning så kan parvis testning prestera likvärdigt. En sak märktes dock under studien som inte var en del av den ursprungliga omfattningen, och det var styrkan som utforskande testning har som ett läroverktyg. / Software testing has been a subject of research since the introduction of computers. Today, when computers and microprocessors are increasingly integrated into the products surrounding us in our daily lives, the importance of effective and accurate software testing increases. Previous research shows that exploratory testing is an effective method for detecting software bugs. Still, with the time and resource aspects considered, there are other potentially more time and resource-effective methods. One such possible method is the pairwise testing method. The literature also shows that this method is effective for finding software bugs. There is, however, not that much research about the comparison of these two methods, which is why this study was conducted. This explanatory case study evaluates how exploratory testing performs compared with pairwise testing. Aspects considered in the evaluation process were the number of detected defects, the severity distribution of the found defects, and what types of defects were found. The data was collected through surveys, interviews, participant observations, and direct observations. With the data collected, it was concluded that the answers to the research questions of this study are ambiguous. There are benefits with both of the techniques, and depending on the exploratory testing conditions, the pairwise technique can perform comparably as the exploratory testing. However, one thing noticed during the study that was not part of the original scope was the strength of exploratory testing as a learning tool. Lastly, some hypotheses were stated, supported by the collected data.