In this thesis, we specifically focus on testing non-functional proprieties. We target the Worst-Case Execution Time (WCET), which is very important for real-time tasks. This thesis applies the concept of targeted mutation, where the mutations are applied to the parts of the code that are most likely to significantly affect the execution time. Moreover, program slicing is used to direct the mutations to the code parts that are likely to have the strongest influence on execution time. The main contribution of the thesis is to implement the method for the experimental evaluation of the targeted mutation testing. This thesis confirms the relevance of the concept of targeted mutation by showing that targeted mutation enables the design of effective test suites for WCET estimation and improves the efficiency of this process by reducing the number of mutants. To experiment the method, we used two C benchmarks from Mälardalen University benchmarks, The first benchmark is Janne-complex code, it consists of 20 line of code and 10 test cases, This benchmark contains two loops, where the inner loop max number of iterations depends on the outer loop current iterations. The results correspond to something Janne flow-analysis should produce. Where the second benchmark is Ludcmp, it consists of 50 lines of code and 9 test cases. This benchmark is a simultaneous linear equation by LU decomposition; it contains two arrays for input and one array for output. However, the number of equations determined by the variable (n). However, both benchmarks LOC decreased after slicing, and each test case applied ten times against each mutant to eliminate the effect of cache and other applications currently running since our experiments depend on estimation of the WCET. The thesis confirms the relevance of the concept of targeted mutation by showing that targeted mutation (i) encourages the design of effective test suites for WCET estimation and (ii) improves the efficiency of this process by reducing the number of mutants.

This work describes different methods of fault injection, the principles of system calls in operating system Linux and the Systemtap tool. The main aim of thesis is design and implementation of the application that work on the principle of fault injection into the system calls using Systemtap tool. Implemented application automatically identifies the existing system calls with the error return values, which are then replaced the original system calls return value.

Context. Software systems are under constant updating for being faulty and to improve and introduce features. The Software testing is the most commonly used  method for validating the quality of software systems. Agile processes help to  automate testing process. A regression test is the main strategy used in testing. Regression testing is time consuming, but with increase in codebases is making it more time extensive and time consuming. Making regression testing time efficient for continuous integration is the new strategy.   Objectives. This thesis focuses on co-relating code packages to test packages by automating mutation to inject error into C code. Regression testing against mutated code establishes co-relations. Co-relation data of particular modified code packages can be used for test selections. This method is most effective than the traditional test selection method. For this thesis to reduce the mutation costs selective mutation method is selected. Demonstrating the proof of concept helps to prove proposed  hypothesis.   Methods. An experiment answers the research questions. Testing of hypothesis on open source C programs will evaluate efficiency. Using this correlation method testers can reduce the testing cycles regardless of test environments. Results. Experimenting with sample programs using automated selective mutation the efficiency to co-relate tests to code packages was 93.4%.   Results. After experimenting with sample programs using automated selective mutation the efficiency to co-relate tests to code packages was 93.4%.   Conclusions. This research concludes that the automated mutation to obtain test selection statistics can be adopted. Though it is difficult for mutants to fail every test case, supposing that this method works with 93.4% efficient test failure on an average, then this method can reduce the test suite size to 5% for the particular modified code package.

Behovet av tillförlitliga, motståndskraftiga, och beständiga system är uppenbart i vårt samhälle, som i ökande grad blir allt mer beroende av mjukvarulösningar. För att uppnå tillfredsställande nivåer av säkerhet och robusthet måste alla system kontinuerligt genomgå tester. En av de största utmaningarna vid automatisering av programvarutestning är avsaknaden av tillförlitliga orakel kapabla att ge korrekta bedömningar av testfall. Metamorfisk testning är en metod som har visats möjlig att applicera för automatisering av testning, men som däremot kräver identifiering av metamorfiska relationer. Det har gjorts försök att identifiera metamorfiska relationer med hjälp av vissa maskininlärningsmodellers förmåga till mönsterigenkänning. Ett stort problem för sådana tillvägagångssätt är mängden tillgängliga och användbara data som dessa ML-modeller kan tränas på. Det huvudsakliga bidraget denna uppsats levererar är en automatiserad metod för att genomföra utökning av data genom källkodsmutation i syfte att skala befintliga datamängder. Specifikt behandlar denna uppsats producering av icke-ekvivalenta mutanter och deras inverkan på maskininlärningsassisterad identifiering av metamorfiska relationer. Resultaten visar att icke-ekvivalenta mutanter kan genereras effektivt, även om manuell granskning är nödvändig för att härleda korrekta etiketter för varje datapunkt. Det visas också att icke-ekvivalenta mutanter kan påverka klassificeringsprestandan positivt i vissa fall, även om resultaten varierar beroende på mutationsoperator och behandlad metamorfisk relation. Framgångsrika framsteg inom testautomatisering kan potentiellt påverka nuvarande standarder för programvaruutveckling genom att förbättra programvarutestningspraxis. Därmed bidrar denna studie till diskussionen om hur automatiserad programvarutestning kan påverka organisationens prestationsförmåga i ett bredare perspektiv. Diskussionen baseras på ramverket för balanserade styrkort, och slutsatsen visar att testautomatisering kan generera fördelaktiga resultat på flera fronter. Det är dock viktigt att samordna sådana initiativ med organisationens strategiska inriktning och långsiktiga mål. / The need for reliable, resilient, and persistent systems is evident in our society, which is becoming increasingly more dependent on software solutions. In order to achieve satisfactory levels of security and robustness, all systems continuously need to undergo testing to detect faults and unwanted functionality. One of the mayor issues in automating software testing is the lack of reliable oracles capable of deriving test case verdicts. Metamorphic testing has been identified as a testing technique which can be used for test automation, though it requires the identification of metamorphic relations. There have been attempts at identifying metamorphic relations using the pattern recognition capabilities of certain machine learning models. A significant problem for any such approach is obtaining a sufficiently large labeled dataset which the ML models can be trained on. The main contribution of this paper is an automated approach to performing data augmentation through a process of source code mutation with the aim of scaling existing datasets. Specifically, this paper considers the generation of non-equivalent mutants and their impact on machine learning assisted identification of metamorphic relations. The results show that non-equivalent mutants can be efficiently generated, although manual oversight is necessary to derive accurate labels for each sample. It is also shown that non-equivalent mutants can positively impact the classification performance in certain instances, though results vary depending mutation operator and considered metamorphic relation. Furthermore, successful advances in the area of test automation can potentially affect current software development standards by improving software testing practices. As such, this study adds to the discussion on how automated software testing might affect organizational performance. The discussion is based on the balanced scorecard framework, and the discussion concludes that test automation can generate beneficial performance outcomes. However, it is imperative to aligning such endeavours with the strategic direction and long-term objectives of the organization.

Software testing is important, but judging whether a set of software tests is effective is difficult. This problem also appears in the classroom as educators more frequently include software testing activities in programming assignments. The most common measures used to assess student-written software tests are coverage criteria—tracking how much of the student’s code (in terms of statements, or branches) is exercised by the corresponding tests. However, coverage criteria have limitations and sometimes overestimate the true quality of the tests. This dissertation investigates alternative measures of test quality based on how many defects the tests can detect either from code written by other students—all-pairs execution—or from artificially injected changes—mutation analysis. We also investigate a new potential measure called checked code coverage that calculates coverage from the dynamic backward slices of test oracles, i.e. all statements that contribute to the checked result of any test. Adoption of these alternative approaches in automated classroom grading systems require overcoming a number of technical challenges. This research addresses these challenges and experimentally compares different methods in terms of how well they predict defect-detection capabilities of student-written tests when run against over 36,500 known, authentic, human-written errors. For data collection, we use CS2 assignments and evaluate students’ tests with 10 different measures—all-pairs execution, mutation testing with four different sets of mutation operators, checked code coverage, and four coverage criteria. Experimental results encompassing 1,971,073 test runs show that all-pairs execution is the most accurate predictor of the underlying defect-detection capability of a test suite. The second best predictor is mutation analysis with the statement deletion operator. Further, no strong correlation was found between defect-detection capability and coverage measures. / Ph. D.

Concurrent programming has become an essential paradigm for reductions in the computational time in many application domains. However, the validation and testing activity is more complex than the testing for sequential programs due to the non-determinism, synchronization and inter-process communication. Mutation testing is based on mistakes produced by software developers and presents a high effectiveness to reveal faults. However, high computational costs limit its applicability even for sequential code, becoming higher for concurrent programs in which each test has to be executed with different (ideally all) thread schedules. To date, only selective mutation have been applied to reduce the number of mutants in concurrent programs, however, the problem of state explosion of thread schedules still remains. This Ph.D. thesis presents the SBBMuT approach that applies deterministic execution and genetic algorithm for the generation and execution of a set of synchronization sequences during the mutation testing of Java multithreaded programs. An experimental study was conducted, and the results showed that the set of synchronization sequences generated by SBBMuT achieved a higher mutation score in comparison with the use of the Java PathFinder model checking tool. / A programação concorrente tornou-se um paradigma essencial para a redução no tempo computacional em muitos domínios de aplicação. No entanto, as atividades de verificação, validação e teste são mais complexas do que o teste para programas sequenciais devido ao não determinismo, sincronização e comunicação entre processos ou threads. O teste de mutação é baseado em enganos cometidos por desenvolvedores de software e apresenta uma alta eficácia para revelar defeitos. No entanto, o alto custo computacional limita a sua aplicação mesmo para programas sequenciais, e tornando-se maior para programas concorrentes no qual cada teste deve ser executado com diferentes (idealmente todas) sequências de sincronizações. Na literatura, apenas mutação seletiva foi aplicada para reduzir o número de mutantes em programas concorrentes, no entanto, o problema de explosão no número de sequências de sincronização ainda permanece. Esta tese de doutorado apresenta a abordagem SBBMuT que aplica execução determinística e algoritmo genético para a geração e execução de um conjunto de sequências de sincronização durante o teste de mutação para programas Java multithread. Um estudo experimental foi conduzido e os resultados mostram que o conjunto de sequências de sincronização gerada pela SBBMuT conseguiu alcançar um escore de mutação maior em comparação com a utilização da ferramenta de validação de modelos Java PathFinder.

Code faults can seriously degrade scientific software accuracy. Therefore, it is imperative that scientific software developers scrutinize their codes in an attempt to find these faults. This thesis explores, particularly, the efficacy of code testing as a method of scientific software code fault detection. Software engineers, as experts in code quality, have developed many code testing techniques, but many of these techniques cannot readily be applied to scientific codes for at least two reasons. First, scientific software testers do not usually have access to an ideal oracle. Second, scientific software outputs, by nature, can only be judged for accuracy and not correctness. Testing techniques developed under the assumption that these two problems can be ignored--as most have been--are of questionable value to computational scientists. To demonstrate the reality of these problems and to provide an example of how software engineers and scientists can begin to address them, this thesis discusses the development and application of a novel technique: Mutation Sensitivity Testing (MST). MST is based on traditional mutation testing, but--in place of a focus on mutant "killing"--MST focuses on assessing the mutation sensitivity of a test set. In this thesis, MST experiments are conducted using eight small numerical routines, four classes of mutation operators, and 1155 tests. The results are discussed and some conclusions are drawn. Two observations are of particular interest to computational scientists. First, it is found that oracles that exhibit uncertainties greater than (approximately) 80% of the expected output are of questionable value when they are used in the testing of scientific software. Second, it is found that a small number of carefully selected tests may be sufficient to falsify a code. / Thesis (Master, Computing) -- Queen's University, 2009-04-19 13:34:08.943

Context: The goal of this research is to form a correlation between code packages and test cases which is done by using automated weak mutation. The correlations formed is used as the statistical test data for selecting relevant tests from the test suite which decreases the size of the test suite and speed up the process. Objectives: In this study, we have done an investigation of existing methods for reducing the computational cost of automatic mutation testing. After the investigation, we build an open source automatic mutation tool that mutates the source code to run on the test cases of the mutated code that maps the failed test to the part of the code that was changed. The failed test cases give the correlation between the test and the source code which is collected as data for future use of the test selection. Methods: Literature review and Experimentation is chosen for this research. It was a controlled experiment done at the Swedish ICT company to mutate the camera codes and test them using the regression test suite. The camera codes provided are from the continuous integration of historical data. We have chosen experimentation as our research because as this method of research is more focused on analyzing the data and implementing a tool using historical data. A literature review is done to know what kind of mutation testing reduces the computational cost of the testing process. The implementation of this process is done by using experimentation Results: The comparative results obtained after mutating the source code with regular mutants and weak mutants we have found that regular mutants and weak mutants are compared with their correlation accuracy and we found that on regular mutation operators we got 62.1% correlation accuracy and coming to weak mutation operators we got 85% of the correlation accuracy. Conclusions: This research on experimentation to form the correlations in generating test selection statistics using automated mutation testing in the continuous integration environment for improving test cases selection in regression testing

Context: Mutation testing is a fault-based technique for measuring the effectiveness of a test set in terms of its ability to detect faults. Mutation testing seeds artificial faults into an application and checks whether a test suite can detect them. If these faults are not found, the test suite is still not considered to be 'good enough'. However, there are also mutations which keep the program semantics unchanged and thus cannot be detected by any test suite. Finding a way to assess these mutations is also known as the equivalent mutant problem (EMP). Objectives: The main objective of this thesis is to conduct a systematic literature review in the field of mutation testing, to identify and classify existing methods for equivalent mutants detection. In addiction, other objectives are: analyze possibilities to improve existing methods for equivalent mutant detection, implement new or improved method and compare it with existing ones. Methods: Based on the systematic literature review method we have went over publications from six electronic databases and one conference proceedings. Standard method was extended by scanning lists of references and some alternative sources: searching in Google Scholar, checking personal websites of relevant authors and contacting all of them. We have performed all the systematic literature review steps such as the protocol development, initial selection, final selection, quality assessment, data extraction and data synthesis. In the second part of this thesis - an experiment, we have implemented four second order mutation testing strategies and compared them each other from four different perspectives: mutants reduction, equivalent mutants reduction, fault detection loss and mutation testing process time reduction. Results: The search identified 17 relevant techniques in 22 articles. Three categories of techniques can be distinguished: detecting (DEM), suggesting (SEM) and avoiding equivalent mutants generation (AEMG). Furthermore, for each technique current state of development and some ideas on how to improve it are provided. The experiment proved that DifferentOperators strategy gives the best results in all four investigated areas. In addition, time for manual mutants classification against equivalence was measured. Assessing one first order mutant takes 11 minutes 49 seconds, while for the second order mutants classification time is 9 minutes 36 seconds in average. Conclusions: After three decades of studies, results obtained for techniques from the DEM group are still far from perfection (best one is detecting 47,63% of equivalent mutants). Thus, new paths for the solution have been developed - SEM and AEMG group. Methods from both categories help in dealing with EMP, however from SEM provide only mutants likely to equivalent, while from AEMG cause some loss of test effectiveness. The conclusion from the experiment is that DifferentOperators strategy gives the best results among all proposed.

Mutation testing is a powerful, but resource intense technique for asserting software quality. This report investigates two claims about one of the mutation operators on procedural logic, the relation operator replacement (ROR). The constrained ROR mutant operator is a type of constrained mutation, which targets to lower the number of mutants as a “do smarter” approach, making mutation testing more suitable for industrial use. The findings in the report shows that the hypothesis on subsumption is rejected if mutants are to be detected on function return values. The second hypothesis stating that a test case can only detect a single top-level mutant in a subsumption graph is also rejected. The report presents a comprehensive overview on the domain of mutation testing, displays examples of the masking behaviour previously not described in the field of mutation testing, and discusses the importance of the granularity where the mutants should be detected under execution. The contribution is based on literature survey and experiment. The empirical findings as well as the implications are discussed in this master dissertation.