In this thesis we present a language and technique to facilitate model-based testing. The core of our approach is an xUnit-like language that allows tests to refer to model entities such as associations. This language can be used by developers to describe tests based on an existing UML model. The tests might even be written before creating a UML model, and be based on requirements. The testing language, including its parser and generators, is written entirely in Umple, an open-source textual modeling tool with semantics closely based on UML, and which generates Java, PHP and several other target languages. Tests in our language can be embedded in Umple or in standalone files. The test language compiler converts our abstract testing language into JUnit, PHPUnit and other domain-language testing environments. In addition to allowing developers to write tests manually, we have created generators that create abstract tests for any Umple model. These generators can be used to verify the Umple compiler and to give Umple users extra confidence in their models. User-defined tests can be standalone or embedded in methods; they can be generic, referring to metamodel elements. Tests can also be located in traits or mixsets to allow testing of separate concerns or product lines. To test our language and the tests written in it, we have created an extensive test suite. We have also implemented mutation testing, that enables varying of features of the models to ensure that runs of the pre-mutation tests then fail.

This dissertation introduces Pikse, a novel methodology for more effective and efficient checking of code conformance to specifications using parallel and incremental techniques, describes a prototype implementation that embodies the methodology, and presents experiments that demonstrate its efficacy. Pikse has at its foundation a well-studied approach -- systematic constraint-driven analysis -- that has two common forms: (1) constraint-based testing -- where logical constraints that define desired inputs and expected program behavior are used for test input generation and correctness checking, say to perform black-box testing; and (2) symbolic execution -- where a systematic exploration of (bounded) program paths using symbolic input values is used to check properties of program behavior, say to perform white-box testing. Our insight at the heart of Pikse is that for certain path-based analyses, (1) the state of a run of the analysis can be encoded compactly, which provides a basis for parallel techniques that have low communication overhead; and (2) iterations performed by the analysis have commonalities, which provides the basis for incremental techniques that re-use results of computations common to successive iterations. We embody our insight into a suite of parallel and incremental techniques that enable more effective and efficient constraint-driven analysis. Moreover, our techniques work in tandem, for example, for combined black-box constraint-based input generation with white-box symbolic execution. We present a series of experiments to evaluate our techniques. Experimental results show Pikse enables significant speedups over previous state-of-the-art. / text

Automated Test Generation (ATG) has been successfully applied in many domains. For the modeling and simulation language Simulink, there has been research on developing tools for ATG with promising results. However, most tools developed as part of academic research and are not publicly available, or severely limited in their ability to be integrated into an industrial workflow. There are commercial ATG tools for Simulink, with Simulink Design Verifier (SLDV) as the de-facto standard tool. For this thesis, we perform an empirical comparison of manual tests to those generated by SLDV. For the comparison, we used 180 components from the propulsion control software developed by our industry partner. All except two components are compatible for test generation to some extent. The majority of components are partially compatible, requiring block replacement or stubbing. Approximation of floating-point numbers is the primary reason for block replacement, which can be performed automatically by SLDV. Two components were incompatible, and 14 required full stubbing of blocks. Using a pre-processing step, the generated tests achieve similar coverage as the manual tests. We performed a Mann–Whitney U test with the hypothesis that the generated tests achieve higher coverage than the manual tests. There are no statistically significant differences for either decision coverage (0.0719), or condition coverage (0.8357). However, for Modified Condition/Decision Coverage, the generated tests achieve higher coverage, and the difference is significant (0.0027). The limitations of ATG were explored by looking at the cases where the generated tests achieved lower coverage than the manual test. We found that the use of floating-point arithmetic and temporal logic increases the time required for test generation, and causes the analysis to hit the time limit. The test generation does not support all custom S-functions and perform stubbing of these blocks. This made the tool unable to reason about persistent storage. Configuration constants have limited support, which was the reason for the coverage difference in three cases. We have concluded that while much effort is required for custom tooling and initial setup, ATG can prove useful for early fault detection in an industrial workflow. ATG would prove especially useful in an automated continuous integration workflow for integration-level conformance testing.

The goal of this thesis is the development of support for test generation and animation based on object-oriented specifications. We aim particularly to take advantage of state-of-the-art satisfiability solving techniques by using an appropriate representation of object-oriented data. While automated test generation seeks a large set of data to execute an implementation on, animation performs computations that comply with a specification based on user-provided input data. Animation is a valuable technique for validating specifications.As a foundation of this work, we present clarifications and a partial formalization of the Object Constraint Language (OCL) as well as some extensions in order to allow for test generation and animation based on OCL specifications.For test generation, we have implemented several enhancements to HOL-TestGen, a tool built on top of the Isabelle theorem proving system that generates tests from specifications in Higher-Order Logic (HOL). We show how SMT solvers can be used to solve various types of constraints in HOL and present a modular approach to case splitting for deriving test cases. The latter facilitates the introduction of splitting rules that are tailored to object-oriented specifications.For animation, we implemented the tool OCLexec for animating OCL specifications. OCLexec generates from operation contracts corresponding Java implementations that call an SMT-based constraint solver at runtime.

The technological development is enabling the production of increasingly complex electronic systems. All such systems must be verified and tested to guarantee their correct behavior. As the complexity grows, testing has become one of the most significant factors that contribute to the total development cost. In recent years, we have also witnessed the inadequacy of the established testing methods, most of which are based on low-level representations of the hardware circuits. Therefore, more work has to be done at abstraction levels higher than the classical gate and register-transfer levels. At the same time, the automatic test equipment based solutions have failed to deliver the required test quality. As a result, alternative testing methods have been studied, which has led to the development of built-in self-test (BIST) techniques. In this thesis, we present a novel hybrid BIST technique that addresses several areas where classical BIST methods have shortcomings. The technique makes use of both pseudorandom and deterministic testing methods, and is devised in particular for testing modern systems-on-chip. One of the main contributions of this thesis is a set of optimization methods to reduce the hybrid test cost while not sacrificing test quality. We have devel oped several optimization algorithms for different hybrid BIST architectures and design constraints. In addition, we have developed hybrid BIST scheduling methods for an abort-on-first-fail strategy, and proposed a method for energy reduction for hybrid BIST. Devising an efficient BIST approach requires different design modifications, such as insertion of scan paths as well as test pattern generators and signature analyzers. These modifications require careful testability analysis of the original design. In the latter part of this thesis, we propose a novel hierarchical test generation algorithm that can be used not only for manufacturing tests but also for testability analysis. We have also investigated the possibilities of generating test vectors at the early stages of the design cycle, starting directly from the behavioral description and with limited knowledge about the final implementation. Experiments, based on benchmark examples and industrial designs, have been carried out to demonstrate the usefulness and efficiency of the proposed methodologies and techniques.

Tests for detecting faults in analog and mixed-signal circuits have been traditionally derived from the datasheet speci and #64257;cations. Although these speci and #64257;cations describe important aspects of the device, in many cases these application oriented tests are costly to implement and are inefficient in determining product quality. Increasingly, the gap between speci and #64257;cation test requirements and the capabilities of test equipment has been widening. In this work, a systematic method to generate and evaluate alternate tests for detecting parametric faults is proposed. We recognize that certain aspects of analog test generation problem are not amenable to automation. Additionally, functional features of analog circuits are widely varied and cannot be assumed by the test generator. To overcome these problems, an extended device under test (DUT) model is developed that encapsulates the DUT and the DUT speci and #64257;c tasks. The interface of this model provides a well de and #64257;ned and uniform view of a large class of devices. This permits several simpli and #64257;cations in the test generator. The test generator is uses a search-based procedure that requires evaluation of a large number of candidate tests. Test evaluation is expensive because of complex fault models and slow fault simulation techniques. A tester-resident test evaluation technique is developed to address this issue. This method is not limited by simulation complexity nor does it require an explicit fault model. Making use of these two developments, an efficient and automated test generation method is developed. Theoretical development and a number of examples are used to illustrate various concepts that are presented in this thesis.

<p>This master thesis investigates how well suited the model-based testing process is for testing a new feature of a Base Station Controller. In model-based testing the tester designs a behavioral model of the system under test, or some part of the system. This model is then given to a test generation tool that will analyze the model and produce interesting test cases. These test cases can either be run on the system in an automatic or manual way depending on what type of setup there is.</p><p>In this report it is suggested that the behavioral model should be produced in as early a stage as possible and that it should be a collaboration between the test team and the design team.</p><p>The advantages with the model-based testing process are a better overview of the test cases, the test cases are always up to date, it helps in finding errors or contradictions in requirements and it performs closer collaboration between the test team and the design team. The disadvantages with model-based testing process are that it introduces more sources where an error can occur. The behavioral model can have errors, the layer between the model and the generated test cases can have errors and the layer between the test cases and the system under test can have errors. This report also indicates that the time needed for testing will be longer compared with manual testing.</p><p>During the pilot, when a part of a new feature was tested, of this master thesis a test generation tool called Qtronic was used. This tool solves a very challenging task which is generating test cases from a general behavioral model and with a good result. This tool provides many good things but it also has its shortages. One of the biggest shortages is the debugging of the model for finding errors. This step is very time consuming because it requires that a test case generation is performed on the whole model. When there is a fault in the model then this test generation can take very long time, before the tool decides that it is impossible to cover the model.</p><p>Under the circumstances that the Qtronic tool is improved on varies issues suggested in the thesis, one of the most important issues is to do something about the long debugging time needed, then the next step can be to use model-based testing in a larger evaluation project at BSC Design, Ericsson.</p>

Šio darbo tikslas yra realizuoti ir ištirti funkcinių vėlinimo testų generavimo algoritmus. Gaminant programuojamuosius įrenginius visada atsiranda galimybė jog gali atsirasti vėlinimo klaidų ( įėjimo signalas dėl kažkokių priežasčių išeina vėliau nei yra numatyta ). Šiuo metu lustai naudojami civilinėje bei karinėje pramonėje įvairiems gamybos procesams valdyti, pvz: medicinoje. Vėlinimo gedimų testavimai yra labai reikšmingi tiek sistemų saugumo užtikrinime, tiek sistemos patikimumo atžvilgiu. Šiais laikais tokie reikalavimai yra keliami vis didesni, nes žmogaus saugumas yra visų svarbiausia. Realizuojamas funkcinis vėlinimo testo sudarymo algoritmas pagal pokyčius išėjimuose, kuris remiasi tik programinio prototipo pirminių įėjimų ir pirminių išėjimų reikšmėmis. Sukūrus sistemą, buvo atliktas eksperimentinis jos tyrimas. Gauti panaudoto algoritmo rezultatai yra patenkinami, kurie yra pateikti šio dokumento priede. Eksperimento metu atliekami testinių rinkinių generavimai 24 loginėms schemoms. Darbe yra 42 panaudoti paveikslėliai, 27 lentel��s bei santrumpų žodynas. Medžiaga surinkta iš penkiolikos literatūros šaltinių. / The object of this work is to research functional test generation methods for delay faults. Manufacturing of programmable chips always has possibility that in those systems will be delay faults that means: input signal for some reasons appear in the output after longer time than the time was definite. In our days, programmable chips are used in the industry for the management of various civil and military manufacturing processes, for instance medicine. Delay fault testing is very important part for the system safety and trustiness. Today those requests become higher and higher, because human safety is common importance. Created test generation algorithm of the functional test that is based solely on the primary input values and the primary output values of the programming prototype. System was tested by simple step by step model,(which is presented in the fourth part of this document). The obtained results are useful and acceptable. All results are presented in the appendix of this document. The experiment contains 24 logic schemas processed test generation algorithm. The work includes of 42 pictures, 27 tables and conceptual dictionary. 15 bibliographical sources have been used.

In the field of networking, testing of communication protocols is an important activity to validate protocol applications before commercialisation. Generally, the services that must be provided by a protocol are described in its specification(s). A specification is generally a standard defined by standards bodies such as ISO (International Standards Organization), IETF (Internet Engineering Task Force), ITU (International Telecommunication Union), etc. The purpose of testing is to verify that the protocol implementations work correctly and guarantee the quality of the services in order to meet customers expectations. To achieve this goal, a variety of testing methods have been developed. Among them, interoperability testing is to verify that several network components cooperate correctly and provide expected services. Conformance testing verifies that a product conforms to its specification. Robustness testing determines the degree to which a system operates correctly in the presence of exceptional inputs or stressful environmental conditions. In this thesis, we focus on interoperability testing. The general architecture of interoperability testing involves a system under test (SUT), which consists of at least two implementations under test (IUT). The objectives of interoperability testing are to ensure that interconnected protocol implementations are able to interact correctly and, during their interaction, provide the services predefined in their specifications. In general, the methods of interoperability testing can be classified into two approaches: active and passive testing. Among them, active test is the most conventionally used technique, which aims to test the implementations (IUT) by injecting a series of test messages (stimuli) and observing the corresponding outputs. However, the intrusive nature of active testing is that the tester has the ability to control IUTS. This implies that the tester interrupts inevitably the normal operations of the system under test. In this sense, active testing is not a suitable technique for interoperability testing, which is often carried out in operational networks. In such context, it is difficult to insert arbitrary testing messages without affecting the normal behavior and the services of the system. On the contrary, passive testing is a technique based only on observation. The tester does not need to interact with the SUT. This allows the test to be carried out without disturbing the normal operations of the system under test. Besides, passive testing also has other advantages such as: for embedded systems to which the tester does not have direct access, test can still be performed by collecting the execution traces of the system and then detect errors by comparing the trace with the behavior of the system described in its specification. In addition, passive testing makes it possible to moniter a system over a long period, and report abnomality at any time.

Software testing is crucial for uncovering software defects and ensuring software reliability. Symbolic execution has been utilized for automatic test generation to improve testing effectiveness. However, existing test generation techniques based on symbolic execution fail to take full advantage of programs’ rich amount of documentation specifying their input constraints, which can further enhance the effectiveness of test generation. In this paper we present a general approach, Document-Assisted Symbolic Execution (DASE), to improve automated test generation and bug detection. DASE leverages natural language processing techniques and heuristics to analyze programs’ readily available documentation and extract input constraints. The input constraints are then used as pruning criteria; inputs far from being valid are trimmed off. In this way, DASE guides symbolic execution to focus on those inputs that are semantically more important. We evaluated DASE on 88 programs from 5 mature real-world software suites: GNU Coreutils, GNU findutils, GNU grep, GNU Binutils, and elftoolchain. Compared to symbolic execution without input constraints, DASE increases line coverage, branch coverage, and call coverage by 5.27–22.10%, 5.83–21.25% and 2.81–21.43% respectively. In addition, DASE detected 13 previously unknown bugs, 6 of which have already been confirmed by the developers.