Context. Software systems are under constant updating for being faulty and to improve and introduce features. The Software testing is the most commonly used  method for validating the quality of software systems. Agile processes help to  automate testing process. A regression test is the main strategy used in testing. Regression testing is time consuming, but with increase in codebases is making it more time extensive and time consuming. Making regression testing time efficient for continuous integration is the new strategy.   Objectives. This thesis focuses on co-relating code packages to test packages by automating mutation to inject error into C code. Regression testing against mutated code establishes co-relations. Co-relation data of particular modified code packages can be used for test selections. This method is most effective than the traditional test selection method. For this thesis to reduce the mutation costs selective mutation method is selected. Demonstrating the proof of concept helps to prove proposed  hypothesis.   Methods. An experiment answers the research questions. Testing of hypothesis on open source C programs will evaluate efficiency. Using this correlation method testers can reduce the testing cycles regardless of test environments. Results. Experimenting with sample programs using automated selective mutation the efficiency to co-relate tests to code packages was 93.4%.   Results. After experimenting with sample programs using automated selective mutation the efficiency to co-relate tests to code packages was 93.4%.   Conclusions. This research concludes that the automated mutation to obtain test selection statistics can be adopted. Though it is difficult for mutants to fail every test case, supposing that this method works with 93.4% efficient test failure on an average, then this method can reduce the test suite size to 5% for the particular modified code package.

The company Tietoevry in Karlstad uses regression testing when working with networks. When a test fails, manual modifications to a command in a text file must be done and a re-test must be performed by executing the command. The process of retesting is both time consuming and error prone. Furthermore, as the number of re-testing processes is constantly increasing, Tietoevry needs a graphical user interface (GUI) to help testers and facilitate the re-testing process. The project’s main aim was to create a GUI that makes re-testing processes less time consuming and less error prone. The project has resulted in a GUI that will serve as a valuable tool for testers. It allows them to conveniently alter commands without the need for manual modifications to text files. The GUI will significantly reduce the time required for the re-testing processes, while also reducing errors.

The validation approach for assuring quality of software does often include the conduction of tests. Software testing includes a wide range of methodology depending on the system level and the component under test. Graphical user interface (GUI) testing consists of high level tests that assert that functions and design element in user interfaces work as expected. The research conducted in this paper focused on GUI testing of web based applications and the movement towards automated testing within the software industry. The question which formed the basis for the study was the following: How should a generic test automation framework be designed in order to allow maintenance between developers and non-developers? The study was conducted on a Swedish consultant company that provides e-commerce web solutions. A work strategy approach for automated testing was identified and an automation framework prototype was produced. The framework was evaluated through a pilot study where testers participated through the creation of a test suite for a specific GUI testing area. Time estimations were collected as well as qualitative measurements through a follow up survey. This paper presents a work strategy proposal for automated tests together with description of the framework system design. The results are presented with a subsequent discussion about the benefits and complexity of creating and introducing automated tests within large scale systems. Future work suggestions are also addressed together with accountancy of the frameworks usefulness for other testing areas besides GUI testing.

This thesis describes an approach to automated regression testing for speech recognition grammars. A prototype Audio Regression Tester called ART has been developed using Microsoft's Speech API and C#. ART allows a user to perform any of three tasks: automatically generate a new XML-based grammar file from standardized SQL database entries, record and cross-reference audio files for use by an underlying speech recognition engine, and perform regression tests with the aid of an oracle grammar. ART takes as input a wave sound file containing speech and a newly created XML grammar file. It then simultaneously executes two tests: one with the wave file and the new grammar file and the other with the wave file and the oracle grammar. The comparison result of the tests is used to determine whether the test was successful or not. This allows rapid exhaustive evaluations of additions to grammar files to guarantee forward process as the complexity of the voice domain grows. The data used in this research to derive results were taken from the LifeLike project. However, the capabilities of ART extend beyond LifeLike. The results gathered have shown that using a person's recorded voice to do regression testing is as effective as having the person do live testing. A cost-benefit analysis, using two published equations, one for Cost and the other for Benefit, was also performed to determine if automated regression testing is really more effective than manual testing. Cost captures the salaries of the engineers who perform regression testing tasks and Benefit captures revenue gains or losses related to changes in product release time. ART had a higher benefit of $21461.08 when compared to manual regression testing which had a benefit of $21393.99. Coupled with its excellent error detection rates, ART has proven to be very efficient and cost-effective in speech grammar creation and refinement.

Inom programvaruutveckling är testning en viktig aktivitet där regressionstest är centralt. Regressionstestning innebär att exekvera om testfall för att säkerställa att de förändringarna som har gjorts i programmet inte har infört nya fel i systemet. Detta görs också med syfte till att kontrollera att systemets befintliga funktioner inte har påverkats negativt av dessa förändringar. Att genomföra regressionstester manuellt är resurskrävande i form av tid och pengar. Därför rekommenderas det att utföra denna aktivitet på ett automatiserat sätt. Ett problem som finns idag med exekvering av automatiserad regressionstest är att testare blir specialiserade på specifika testmiljöer på grund av det används många olika testmiljöer. Därför skulle de inte kunna stödja förbättringen av andra testares arbetsflöde. Således fokuserar denna studie på att beskriva de aktuella arbetssätt inom området exekvering av automatiserade regressionstester samt hur dessa kan utvecklas. Det har varit nödvändigt att genomföra personliga intervjuer samt litteraturstudie för att kunna besvara studiens forskningsfrågor. I detta arbete har det kommits fram till skapandet av en allmän ram för testare att arbeta mer effektivt med genomförandet av automatiserade regressionstestning oavsett testmiljön. Ramverket innehåller 7 faser: (1) Val av testfall, (2) Utföring/Exekvering av tester, (3) Kontroll/Analysera av resultat, (4) Skrivandet av rapport för testresultat, (5) Arkivering av testresultat rapporter, (6) Värdering och tilldelning av uppkommande problem samt (7) Hantering av testcykeln. / Software testing is an important activity within the software development area, where regression testing is essential. Regression testing implies re-running test cases in order to ensure that changes made to the software do not introduce new errors and to guarantee that the system’s functionalities have not been affected by those changes. To execute regression testing in manual mode involves valuable resources, specifically time and money. Therefore it is recommended to carry out this activity in an automated manner. A problem that faces the execution of automated regression testing is that testers are becoming specialized in specific test environments due to the existing diversity of tools used. Therefore testers would not be able improve test processes related to different environments. Thus, this paper focuses on identifying the current working manner within the domain of execution of automated regression testing and to improve it. Then, it has been necessary to conduct personal interviews (7) and a literature study in order to answer the study’s questions. This work provides a general framework for testers to work more effectively with execution of automated regression testing regardless of the test environment. The framework includes 7 stages: (1) Select test cases, (2) Execute/Run tests, (3) Monitoring/Analyzing results, (4) Write a report of the test results, (5) Archive the reports of the test results, (6) Estimate and assign the raised problems, and (7) Manage the test cycle.

Regression testing of an evolving software system may involve significant challenges. While, there would be a requirement of maximising the probability of finding out if the latest changes to the system has broken some existing feature, it needs to be done as economically as possible. A particularly important class of software systems are API libraries. Such libraries would typically constitute a very important component of many software systems. High quality requirements make it imperative to continually optimise the internal implementation of such libraries without affecting the external interface. Therefore, it is preferred to guide the regression testing by some kind of formal specification of the library. The testing problem comprises of three parts: computation of test data, execution of test, and analysis of test results. Current research mostly focuses on the first part. The objective of test data computation is to maximise the probability of uncovering bugs, and to do it with as few test cases as possible. The problem of test data computation for regression testing is to select a subset of the original test suite running which would suffice to test for bugs probably inserted in the modifications done after the last round of testing. A variant of this problem is that of regression testing of API libraries. The regression testing of an API is usually done by making function calls in such a way that the sequence of function calls thus made suffices a test specification. The test specification in turn embodies some concept of completeness. In this thesis, we focus on the problem of test sequence computation for the regression testing of API libraries. At the heart of this method lies the creation of a state space model of the API library by reverse engineering it by executing the system, with guidance from an formal API specification. Once the state space graph is obtained, it is used to compute test sequences for satisfying some test specification. We analyse the theoretical complexity of the problem of test sequence computation and provide various heuristic algorithms for the same. State space explosion is a classical problem encountered whenever there is an attempt of creating a finite state model of a program. Our method also faces this limitation. We explore a simple and intuitive method of ameliorating this problem – by simply reducing the size of the state vector. We develop the theoretical insights into this method. Also, we present experimental results indicating the practical effectiveness of this method. Finally, we bring all this together into the design and implementation of a tool called Modest.

In this thesis, a new modeling technique is proposed for minimizing regression testing effort in software product lines. The &ldquo / Product Flow Model&rdquo / is used for the common representation of products in application engineering and the &ldquo / Domain Service and Object Model&rdquo / represents the variant based relations between products and core assets. This new approach provides a solution for avoiding unnecessary work load of regression testing using the principles of sub-service decomposition and variant based product/sub-service traceability matrices. The proposed model is adapted to a sample product line targeting the banking domain, called Loyalty and Campaign Management System, where loyalty campaigns for credit cards are the products derived from core assets. Reduced regression test scope after the realization of new requirements is demonstrated through a case study. Finally, efficiency improvement in terms of time and effort in the test process with the adaptation of the proposed model is discussed.

Kasdieninis žiniatinklio informacinių sistemų (toliau – IS) naudojimas paprastų vartotojų, verslo ir vyriausybinių organizacijų tarpe, reikalauja vis aukštesnės tokių IS kokybės. Žiniatinklio IS dažniausiai yra tokia programinė įranga, kurios sudėtingumas, dėl keleto apjungiamų technologijų, yra aukštas, o pokyčių greitis yra didelis. Vykdant palaikymo ir priežiūros darbus, gali atsirasti begalė klaidų, kurias iššaukia atlikti pakeitimai. Regresinio testavimo paskirtis yra užtikrinti, jog įdiegus programinės įrangos pakeitimus, tokius kaip naujo funkcionalumo pridėjimas ar jau egzistuojančio taisymas, ankstesnėje IS versijoje veikęs funkcionalumas veiks taip pat gerai ir naujojoje versijoje. Šiame darbe tiriami iš HTML žymų sudarytą IS atsaką galintys palyginti regresinio testavimo algoritmai (toliau – testų orakulai). HTML ãtsako kaip teksto lyginimas pateikia per daug klaidingai teigiamų (angl. false positives) rezultatų – testuotojas užverčiamas pranešimais apie netikras klaidas. Mūsų realizuotas testų orakulas HTML žymų atsaką nagrinėja kaip medžio struktūrą. Toks algoritmo veikimas leidžia tikrinti arba netikrinti aprašytas HTML semantines savybes, ko pasekoje yra sumažinamas klaidingai teigiamų ir klaidingai neigiamų rezultatų kiekis. Realizuotas testų orakulas nuo literatūroje aprašomų ir rinkoje siūlomų testų orakulų skiriasi tuo, jog nepraneša apie klaidą, kai HTML atsakas būna papildytas naujomis žymomis, kurios nekeičia pradinės medžio struktūros. Žiniatinklio IS... [toliau žr. visą tekstą] / Common usage of web information systems among consumers, business and governmental organizations require higher quality of such systems. Web information systems is such a software, which has high complexity because of the multiple technology usage and tends to rapid changes. Errors may occur because of the changes that have been made through maintenance period. Regression testing purpose is to ensure, that software changes such as new functionality or previously existed functionality modification, did not affect functionality, which was well working in the older version. In these master thesis we analyze test oracle comparators which can compare HTML response. HTML as a text comparison produces too many false positives. Meanwhile our test oracle comparator analyses HTML response as is was a tree structure. Thus implementation of the test oracle comparator algorithm enables opportunity to validate or not to validate described HTML semantic features. Therefore, we get fewer false positives and false negatives. Our implemented test oracle comparator differs from the ones described in literature or the ones proposed in market by capability not to warn tester about changes such as newly added HTML tag, which do not change overall tree structure. In this way the introduction of natural functionality growth is allowed. After improvement our test oracle comparator detects 90 % errors with the 90 % precision, whereas in the market existing test oracle comparator of the regression... [to full text]

Context: The goal of this research is to form a correlation between code packages and test cases which is done by using automated weak mutation. The correlations formed is used as the statistical test data for selecting relevant tests from the test suite which decreases the size of the test suite and speed up the process. Objectives: In this study, we have done an investigation of existing methods for reducing the computational cost of automatic mutation testing. After the investigation, we build an open source automatic mutation tool that mutates the source code to run on the test cases of the mutated code that maps the failed test to the part of the code that was changed. The failed test cases give the correlation between the test and the source code which is collected as data for future use of the test selection. Methods: Literature review and Experimentation is chosen for this research. It was a controlled experiment done at the Swedish ICT company to mutate the camera codes and test them using the regression test suite. The camera codes provided are from the continuous integration of historical data. We have chosen experimentation as our research because as this method of research is more focused on analyzing the data and implementing a tool using historical data. A literature review is done to know what kind of mutation testing reduces the computational cost of the testing process. The implementation of this process is done by using experimentation Results: The comparative results obtained after mutating the source code with regular mutants and weak mutants we have found that regular mutants and weak mutants are compared with their correlation accuracy and we found that on regular mutation operators we got 62.1% correlation accuracy and coming to weak mutation operators we got 85% of the correlation accuracy. Conclusions: This research on experimentation to form the correlations in generating test selection statistics using automated mutation testing in the continuous integration environment for improving test cases selection in regression testing

A lot of time is spent on manual software testing. When software can interact with exponentially more and more components the number of tests required will increase at the same rate. Automation of these manual tests has been previously shown that a time-saving can occur with test automation. When tests need to be done on software that interacts with physical components the results of testing automation are harder to predict. This thesis aims to explain the impact of physical components on the development and the result of test tools developed for use on automated tests on objects with physical components. In this work, a test tool for automatic regression testing for height adjustable table legs were developed. The method used to design the tool was Design Science Research where, for example, TESLA was adapted for use in automated tests with physical components. TESLA is a language for specifying test cases and allowing test automation with embedded systems that was implemented in the test tool's design which could be used to automatically generate scripts and automatically executes test cases. With the development and design of the testing tool, the thesis attempted to answer how a test tool should be designed when physical components are included in the test object and how the measurement accuracy of the tool may affect the test result. Experiments have been carried out during the development process where regression tests have been performed according to a test protocol. During these experiments a test protocol is executed. One manually by hand and one is done automatically with the testing tool. The results of the experiments show that the execution of the test protocol execution with the test tool gave a time saving of 35% compared to the manual tests. The physical components of the test object were shown to affect the automatic execution time negative and further analysis of previous research has shown that this result is insufficient to justify the large initial time that is required to automate tests. / Mycket tid spenderas ofta på manuell testning av mjukvara. Då en mjukvara kan interagera med exponentiellt mer komponenter så kommer antalet tester att öka i samma takt. Automatisering av dessa manuella tester har med tidigare forskning bevisat att en tidsbesparing kan ske med testautomatiseringen. Då tester måste utföras på mjukvara som interagerar med fysiska komponenter så är resultaten av testautomatisering inte lika klara och arbetet syftar på att förklara de fysiska komponenternas påverkan på utveckling och resultatet för testverktyg som används till automatisk testning av testobjekt med fysiska komponenter. I detta arbete så framtogs ett testverktyg för automatiska regressionstest för höj och sänkbara bordsben. Metoden som användes för att utveckla verktyget var Design Science Research där till exempel TESLA anpassades för användning i automatiska tester med fysiska komponenter. TESLA är ett språk för att specificera testfall och möjliggöra testautomatisering i inbyggda system implementerades i testverktygets design och kunde då automatisk generera skript och automatiskt exekvera testfall. Med utvecklingen och design av testverktyget så försökte arbetet svara på hur ett testverktyg bör vara designat när fysiska komponenter ingår i testobjektet och hur testverktygets mätnoggrannhet kan påverka testresultatet. Experiment har utförts under arbetets gång där regressionstester har utförts enligt ett testprotokoll. Under experimenten utfördes ett testprotokoll manuellt för hand och ett automatiskt med testverktyget. I båda situationerna så mättes testprotokollets utförande i tid. Testverktyget gav en tidsbesparing på 35% visavi de manuella testerna. Testobjektets fysiska komponenter hade en negativ påverkan på den automatiska exekveringstiden och vidare analys av tidigare forskning har visat att detta resultat inte räcker för att motivera den stora initiala tidsinvestering som krävs för att automatisera tester.