Functional Test of any given system is used to verify that the top level system is performing according to the specifications and all of the sub systems, i.e., hardware, software, inputs, outputs and sensors are operating properly. The term System is used here in context of any device or equipment consisting of hardware, software, sensors, virtual sensors and inputs / outputs. One of the examples of such a system is a semiconductor fabrication equipment. There have only been a few approaches that are used to perform the functional test of a system. Engineers typically develop custom test equipment to interface with the system under test and determine proper functioning of all the sub systems and behavior of overall system. In another method, domain experts, utilize the software of the system under test, and manually actuate / stimulate sub systems and then observe and record outcomes to determine whether the system exhibited correct behavior. A novel solution of a reusable functional test automation framework is developed. The framework interfaces with the system under test via the exposed API, and allow domain experts with minimal or no programming background to create test suites to methodically test overall functionality of the system. / text

Master of Science / Department of Computing and Information Sciences / Robby / This report is a case study of applying Model-Based testing approach using SpecExplorer, which is a model-based testing tool developed by Microsoft, to test a component of Microsoft Network Monitor. The system under test is the UI of the Network Monitor feature, Parser Profiles Management. Model-Based testing is a methodology for automated testing which not only automates the test execution, but the test design and generation. This approach starts by expressing an abstract model of the system which is a smaller subset of the product behavior, but retains essential elements which form the focus of the testing problem. A model-based testing tool creates a finite state machine from the model which is traversed to produce test cases. Thus, it provides more efficient coverage and flexibility in developing and maintaining test cases.

<p>Thorough testing of software is necessary to assure the quality of a product before it is released. The testing process requires substantial resources in software development. Model-based software development provides new possibilities to automate parts of the testing process. By automating tests, valuable time can be saved. This thesis focuses on different ways to utilize models for automatic generation of test vectors and how test coverage analysis can be used to assure the quality of a test suite or to find "dead code" in a model. Different test-automation techniques have been investigated and applied to a model of an adaptive cruise control system (ACC) used at Scania. Source code has been generated automatically from the model, model coverage and code coverage has therefore been compared. The work with this thesis resulted in a new method to create test vectors for models based on a combinatorial test technique.</p>

Today, interoperability is the major challenge for e-Business and e-Government domains. The fundamental solution is the standardization in different levels of business-to-business interactions. However publishing standards alone are not enough to assure interoperability between products of different vendors. In this respect, testing and certification activities are very important to promote standard adoption, validate conformance and interoperability of the products and maintain correct information exchange. In e-Business collaborations, standards need to address different layers of interoperability stack / communication layer, business document layer and business process layer. Although there have been conformance and interoperability testing tools and initiatives for each one of these categories, there is currently no support for testing an integration of the above within a test scenario which is similar to real life use cases. Together with the integration of different layers of testing, testing process should be automated so that test case execution can be done at low cost, and repeated if required. In this theses, a highly adaptable and flexible Test Execution Model and a complementary XML based Test Description Language consisting of high level test constructs which can handle or simulate different parts or layers of the interoperability stack is designed. The computer interpretable test description language allow dynamic set up of test cases and provides flexibility to design, modify, maintain and extend the test functionality in contrast to a priori designed and hard coded test cases. The work presented in this thesis is a part of the TestBATN system supported by TUBITAK, TEYDEB Project No: 7070191.

Testing is the dominating method for quality assurance of industrial software. Despite its importance and the vast amount of resources invested, there are surprisingly limited efforts spent on testing research, and the few industrially applicable results that emerge are rarely adopted by industry. At the same time, the software industry is in dire need of better support for testing its software within the limited time available. Our aim is to provide a better understanding of how test cases are created and applied, and what factors really impact the quality of the actual test. The plethora of test design techniques (TDTs) available makes decisions on how to test a difficult choice. Which techniques should be chosen and where in the software should they be applied? Are there any particular benefits of using a specific TDT? Which techniques are effective? Which can you automate? What is the most beneficial way to do a systematic test of a system? This thesis attempts to answer some of these questions by providing a set of guidelines for test design, including concrete suggestions for how to improve testing of industrial software systems, thereby contributing to an improved overall system quality. The guidelines are based on ten studies on the understanding and use of TDTs. The studies have been performed in a variety of system domains and consider several different aspects of software test. For example, we have investigated some of the common mistakes in creating test cases that can lead to poor and costly testing. We have also compared the effectiveness of different TDTs for different types of systems. One of the key factors for these comparisons is a profound understanding of faults and their propagation in different systems. Furthermore, we introduce a taxonomy for TDTs based on their effectiveness (fault finding ability), efficiency (fault finding rate), and applicability. Our goal is to provide an improved basis for making well-founded decisions regarding software testing, together with a better understanding of the complex process of test design and test case writing. Our guidelines are expected to lead to improvements in testing of complex industrial software, as well as to higher product quality and shorter time to market.

A utilização desta abordagem no ciclo de desenvolvimento de uma aplicação web traz algumas vantagens, como ser necessário gerar manualmente apenas o modelo de comportamento de cada funcionalidade da aplicação, (pois os demais artefatos são gerados automaticamente), consumindo menos tempo e estando menos sujeitos a erros, além de prevenir diferentes interpretações dos requisitos pelos stakeholders, desenvolvedores e testadores. O tempo despendido na especificação dos modelos é compensado pelo tempo economizado com a geração dos cenários e do código de testes. / The testing activity throughout software development is fundamental to the pursuit of software quality and reliability, finding faults to be removed. However, despite its importance, software testing is often an underutilized phase in software development. Moreover, tests are proved to be expensive, difficult and problematic when not done in the appropriate way. A new paradigm for software testing is model-driven testing (MDT), which can be defined as software testing where test cases are derived from a model that describes some aspects of the system being tested, such as behavior, for example. This description, often using UML diagrams and/or its profiles, can be processed to produce a set of test cases. Software specifications based on usage scenarios expressed by appropriate UML diagrams are considered significant and effective, because they describe the system’s requirements from an intuitive and visual perspective. Thus, they can be used for the description of acceptance tests, which validate that the system meets user requirements. These specifications also facilitate the automation of this kind of test. Test automation can decrease time spent on testing, thereby reducing the cost of this activity. Thus, this work proposes an approach for automated generation of acceptance tests from U2TP (the UML 2.0 test profile) diagrams for web applications, based on behavior driven development (BDD) paradigm, obtaining acceptance scenarios and executable test code supported by an acceptance testing automation framework. This approach was applied on an actual development environment, by means of an experiment. Using this approach in an web application development cycle has some advantages, such as being required only to manually generate the model of behavior of each application functionality (because other artifacts are generated automatically), thus being less time consuming and less prone to errors, and preventing different interpretations of requirements by stakeholders, developers and testers. The time spent at the models’ specification is compensated by the time saved with the generation of scenarios and test code.

A utilização desta abordagem no ciclo de desenvolvimento de uma aplicação web traz algumas vantagens, como ser necessário gerar manualmente apenas o modelo de comportamento de cada funcionalidade da aplicação, (pois os demais artefatos são gerados automaticamente), consumindo menos tempo e estando menos sujeitos a erros, além de prevenir diferentes interpretações dos requisitos pelos stakeholders, desenvolvedores e testadores. O tempo despendido na especificação dos modelos é compensado pelo tempo economizado com a geração dos cenários e do código de testes. / The testing activity throughout software development is fundamental to the pursuit of software quality and reliability, finding faults to be removed. However, despite its importance, software testing is often an underutilized phase in software development. Moreover, tests are proved to be expensive, difficult and problematic when not done in the appropriate way. A new paradigm for software testing is model-driven testing (MDT), which can be defined as software testing where test cases are derived from a model that describes some aspects of the system being tested, such as behavior, for example. This description, often using UML diagrams and/or its profiles, can be processed to produce a set of test cases. Software specifications based on usage scenarios expressed by appropriate UML diagrams are considered significant and effective, because they describe the system’s requirements from an intuitive and visual perspective. Thus, they can be used for the description of acceptance tests, which validate that the system meets user requirements. These specifications also facilitate the automation of this kind of test. Test automation can decrease time spent on testing, thereby reducing the cost of this activity. Thus, this work proposes an approach for automated generation of acceptance tests from U2TP (the UML 2.0 test profile) diagrams for web applications, based on behavior driven development (BDD) paradigm, obtaining acceptance scenarios and executable test code supported by an acceptance testing automation framework. This approach was applied on an actual development environment, by means of an experiment. Using this approach in an web application development cycle has some advantages, such as being required only to manually generate the model of behavior of each application functionality (because other artifacts are generated automatically), thus being less time consuming and less prone to errors, and preventing different interpretations of requirements by stakeholders, developers and testers. The time spent at the models’ specification is compensated by the time saved with the generation of scenarios and test code.

The purpose of this thesis is to investigate organizational quality assurance through the international testing standard ISO 29119. The focus will be on how an organization carries out testing processes and designs and implements test cases. Keyword driven testing is a test composition concept in ISO 29119 and suitable for automation. This thesis will answer how keyword driven testing can facilitate the development of maintainable test cases and support test automation in an agile organization. The methodology used was a qualitative case study including semi-structured interviews and focus groups with agile business units within Scania IT. Among the interview participants were developers, test engineers, scrum masters and a unit manager. The results describe testing practices carried out in several agile business units, maintainability issues with test automation and general ideas of how test automation should be approached. Common issues with test automation were test cases failing due to changed test inputs, inexperience with test automation frameworks and lack of resources due to project release cycle. This thesis concludes that keyword driven testing has the potential of solving several maintainability issues with test cases breaking. However, the practicality and effectiveness of said potential remain unanswered. Moreover, successfully developing an automated keyword driven testing framework requires integration with existing test automation tools and considering the agile organizational circumstances.

Thorough testing of software is necessary to assure the quality of a product before it is released. The testing process requires substantial resources in software development. Model-based software development provides new possibilities to automate parts of the testing process. By automating tests, valuable time can be saved. This thesis focuses on different ways to utilize models for automatic generation of test vectors and how test coverage analysis can be used to assure the quality of a test suite or to find "dead code" in a model. Different test-automation techniques have been investigated and applied to a model of an adaptive cruise control system (ACC) used at Scania. Source code has been generated automatically from the model, model coverage and code coverage has therefore been compared. The work with this thesis resulted in a new method to create test vectors for models based on a combinatorial test technique.

A utilização desta abordagem no ciclo de desenvolvimento de uma aplicação web traz algumas vantagens, como ser necessário gerar manualmente apenas o modelo de comportamento de cada funcionalidade da aplicação, (pois os demais artefatos são gerados automaticamente), consumindo menos tempo e estando menos sujeitos a erros, além de prevenir diferentes interpretações dos requisitos pelos stakeholders, desenvolvedores e testadores. O tempo despendido na especificação dos modelos é compensado pelo tempo economizado com a geração dos cenários e do código de testes. / The testing activity throughout software development is fundamental to the pursuit of software quality and reliability, finding faults to be removed. However, despite its importance, software testing is often an underutilized phase in software development. Moreover, tests are proved to be expensive, difficult and problematic when not done in the appropriate way. A new paradigm for software testing is model-driven testing (MDT), which can be defined as software testing where test cases are derived from a model that describes some aspects of the system being tested, such as behavior, for example. This description, often using UML diagrams and/or its profiles, can be processed to produce a set of test cases. Software specifications based on usage scenarios expressed by appropriate UML diagrams are considered significant and effective, because they describe the system’s requirements from an intuitive and visual perspective. Thus, they can be used for the description of acceptance tests, which validate that the system meets user requirements. These specifications also facilitate the automation of this kind of test. Test automation can decrease time spent on testing, thereby reducing the cost of this activity. Thus, this work proposes an approach for automated generation of acceptance tests from U2TP (the UML 2.0 test profile) diagrams for web applications, based on behavior driven development (BDD) paradigm, obtaining acceptance scenarios and executable test code supported by an acceptance testing automation framework. This approach was applied on an actual development environment, by means of an experiment. Using this approach in an web application development cycle has some advantages, such as being required only to manually generate the model of behavior of each application functionality (because other artifacts are generated automatically), thus being less time consuming and less prone to errors, and preventing different interpretations of requirements by stakeholders, developers and testers. The time spent at the models’ specification is compensated by the time saved with the generation of scenarios and test code.