yes / We are pleased to welcome you to the 17th edition of the Advances in Model-Based Testing Workshop (A-MOST 2021), collocated with the IEEE International Conference on Software Testing, Verification and Validation (ICST 2021).

Software testing is a vital part of the software development process and is used for the purposes of quality assurance, reliability estimation and verification and validation. However, software testing is extremely costly and time consuming. For instance, prior research indicated that more than 50% of the software development cost is devoted to testing. Due to the increased use of the Unified Modeling Language (UML) and model driven architecture approach in systems analysis and design, Model-Based Testing has been discussed as a prominent solution for software testing to address the above problems. This study presents a methodology for determining the test path which can then be further used to determine the test case for Model-Based Testing. Three real-world cases are used to test the usability (including the concepts, application, and advantages) of the proposed methodology. With this approach, software errors can be found at the systems analysis and design stage and thereby reduce the cost of software testing and enhance the efficiency of system development.

Yes / We present a formalism to represent cyclic models and study di erent semantic frameworks that support testing. These models combine sequences of observable actions and the passing of (discrete) time and can be used to specify a number of classes of reactive systems, an example being robotic systems. We use implementation relations in order to formally define a notion of correctness of a system under test (SUT) with respect to a specification. As usual, the aim is to devise an extension of the classical ioco implementation relation but available timed variants of ioco are not suitable for cyclic models. This paper thus defines new implementation relations that encapsulate the discrete nature of time and take into account not only the actions that models can perform but also the ones that they can refuse. In addition to defining these relations, we study a number of their properties and provide alternative characterisations, showing that the relations are appropriate conservative extensions of trace containment. Finally, we give test derivation algorithms and prove that they are sound and also are complete in the limit. / Engineering and Physical Sciences Research Council Grant numbers: EP/R025134/2. Ministerio de Economía y Competitividad Grant numbers: RTI2018-093608-B-C31. Comunidad de Madrid Grant numbers: S2018/TCS-4314

Testing is the dominant technique for quality assurance of software systems. It typically consumes considerable resources in development projects, and is often performed in an ad hoc manner. This thesis is concerned with model-based testing, which is an approach to make testing more systematic and more automated. The general idea in model-based testing is to start from a formal model, which captures the intended behavior of the software system to be tested. On the basis of this model, test cases can be generated in a systematic way. Since the model is formal, the generation of test suites can be automated and with adequate tool support one can automatically quantify to which degree they exercise the tested software. Despite the significant improvements on model-based testing in the last 20 years, acceptance by industry has so far been limited. A number of commercially available tools exist, but still most testing in industry relies on manually constructed test cases. This thesis address this problem by presenting a methodology and associated tool support, which is intended to be used for model-based testing of communication protocol implementations in industry. A major goal was to make the developed tool suitable for industrial usage, implying that we had to consider several problems that typically are not addressed by the literature on model-based testing. The thesis presents several technical contributions to the area of model-based testing, including - a new specification language based on the functional programming language Erlang, - a novel technique for specifying coverage criteria for test suite generation, and - a technique for automatically generating test suites. Based on these developments, we have implemented a complete tool chain that generates and executes complete test suites, given a model in our specification language. The thesis also presents a substantial industrial case study, where our technical contributions and the implemented tool chain are evaluated. Findings from the case study include that test suites generated using (model) coverage criteria have at least as good fault-detection capability as equally large random test suites, and that model-based testing could discover faults in previously well-tested software where previous testing had employed a relaxed validation of requirements.

A case study for the tool LBTest illustrating benets and limitations of the tool along the terms of usability, results and costs. The study shows the use of learning based testing on a safety critical application in the avionics industry. While requiring the user to have the oretical knowledge of the tools inner workings, the process of using the tool has benefits in terms of requirement analysis and the possibility of finding design and implementation errors in both the early and late stages of development

Software testing is a vital part of the software development process, usually implemented at the coding stage, and costly. Due to the increased use of the Unified Modeling Language (UML) and Model Driven Architecture (MDA) approach in systems analysis and design, Model-Based Testing has been discussed as a prominent solution for software testing to reduce the cost of software testing. Prior researches proposed an integrated method which utilizes the artifacts from the Platform Independent Model (PIM) to construct the test paths and generate the test cases. This study develops a methodology which extracting the information from PIM (e.g., Sequence Diagram and Class diagram) to generate the test cases directly. The research methodology is articulated using the design science research methodology. A usability evaluation is performed to demonstrate its applicability. With this methodology, the test cases can be easily generated; thereby reducing the cost and enhancing the efficiency of Model-Based Testing.

Software testing is a vital part of the software development process and is costly. Due to the increased use of the unified modeling language and model driven architecture approach in systems analysis and design, model-based testing has been discussed as a prominent solution for software testing to reduce the cost of software testing. Prior researches proposed an integrated method which utilizes the artifacts from the Platform Independent Model (PIM) to construct the test path and constrained class tuples. These two can then be integrated to construct an Integrated Testing Model (ITM) for determining the test data and test cases for Model-Based Testing. This study develops a methodology which extracting the information from ITM to construct the test cases. The research methodology is articulated using the design science research methodology. A prototype embedded methodology has been developed to automatically generate the test cases. A usability evaluation is performed on the prototype to demonstrate its usability. With this methodology, the test cases can be generated automatically; thereby reducing the cost and enhancing the efficiency of Model-Based Testing.

Currently, the increasing complexity of software and the short release cycles are becoming a challenge for testing software in an efficient and effective way. Traditionally, creating tests is done manually by engineers, which are then automatically or manually executed on the actual software. Manually creating test cases is a time-consuming effort. For the last couple of decades, researchers have proposed ways to improve this process by automating parts of the testing steps. One of these approaches that have gained a lot of attention is called Model-Based Testing (MBT). MBT has been suggested as a way of automatically creating tests at a lower cost. Nonetheless, it is not very well studied how MBT is actually applied in industrial contexts and how these tests compare to manually written ones. This is particularly true for industrial control software such as the one found in the train domain, where strict requirements on testing are in place. In this thesis, we investigate the literature and review the related work on case studies on the MBT use in industry and its evaluation. We perform a case study to evaluate MBT on a train control management system provided by Bombardier Transportation. We use Comformiq Creator MBT Tool to create models for functional requirements of a master controller function and generate test cases. We provide the result of the modeling approach as well as a comparison between automatic test cases created by Conformiq Creator and manual test cases written by industrial engineers at Bombardier Transportation using the following metrics: test coverage and time spent on testing. The results of this comparison suggest that test coverage of MBT is higher and test cases are more detailed than manual testing. Our results are not conclusive in regard to the cost of using MBT, mainly because this depends on different testing scenarios and how testing is performed. We show that MBT is a suitable approach for modeling the functional requirements of a realistic industrial control software function. In this thesis work, we focus on system-level testing. As future work, applying MBT on lower levels of testing can be a promising way forward for evaluation. In addition, the transformation of these test cases into executable test scripts and the possible problems needs to be investigated further.

In this thesis we present a language and technique to facilitate model-based testing. The core of our approach is an xUnit-like language that allows tests to refer to model entities such as associations. This language can be used by developers to describe tests based on an existing UML model. The tests might even be written before creating a UML model, and be based on requirements. The testing language, including its parser and generators, is written entirely in Umple, an open-source textual modeling tool with semantics closely based on UML, and which generates Java, PHP and several other target languages. Tests in our language can be embedded in Umple or in standalone files. The test language compiler converts our abstract testing language into JUnit, PHPUnit and other domain-language testing environments. In addition to allowing developers to write tests manually, we have created generators that create abstract tests for any Umple model. These generators can be used to verify the Umple compiler and to give Umple users extra confidence in their models. User-defined tests can be standalone or embedded in methods; they can be generic, referring to metamodel elements. Tests can also be located in traits or mixsets to allow testing of separate concerns or product lines. To test our language and the tests written in it, we have created an extensive test suite. We have also implemented mutation testing, that enables varying of features of the models to ensure that runs of the pre-mutation tests then fail.

Det finns en omfattande litteratur kring modellbaserad testning (MBT) men med få tecken på att metoden har fått något större genomslag i industrin. Målet med studien är att identifiera exempel där MBT används som testmetod inom industrin och eventuella hinder som finns i MBT-processen. För att genomföra detta används en mixed-methods ansats bestående av en systematisk litteraturstudie följt av en utforskande fallstudie. I fallstudien tillämpas MBT med hjälp av verktyget Modbat på ett mjukvarusystem. Endast ett fåtal industriella tillämpningar av MBT identifieras i litteraturstudien. Totalt sju studier kvarstår efter fulltextgranskningen. Studierna finns primärt inom mjukvaruindustrin och flygindustrin men innehåller även exempel från hälso- sjukvård och bilindustrin. Den utforskande fallstudien indikerar tre typer av hinder. Det första är mängden arbete med, samt bristande användarvänlighet hos verktygen. Den andra är svårigheten med att skriva ett adaptionslager som integrerar systemet med verktyget och modellen för att göra testfallen körbara. Det sista hindret är det kraftiga beroendet på att modellen utformas korrekt och stämmer med systemets tilltänkta beteende. Dessa tre hinder pekas även ut i verken från litteraturstudien. Vidare pekas bland annat även icke-tekniska svårigheter ut under litteraturstudien i form av att hela arbetsgruppen och ledningen behöver engageras för att införa ett nytt arbetssätt. Med en begränsad fallstudie och ett enkelt system bekräftas tre hinder i MBT-processen som även identifieras i litteraturgenomgången. MBT framstår som ett primärt akademiskt område med ett fåtal exempel på användning inom industrin. / There is extensive literature concerning model-based testing (MBT) but few signs that the method have had any major breakthrough in the industry. The goal of this study is to identify examples of MBT being used in the industry and any challenges faced during the MBT-process. The study is conducted using a mixed methods approach, consisting of a systematic literature review followed by an exploratory case-study. The case-study applies MBT to a software system using the MBT-tool Modbat.Only seven studies remain after the fulltext review is performed. The studies are primarily from the software and aerospace industries but also include examples from the healthcare and automotive industries. The exploratory case-study identifies three challenges. The first one is the amount of work and lacking usability related to the MBT-tools. The second challenge is implementing the adaption layer, integrating the system under test with the tool and model to make test cases executable. The final challenge is the dependency on a correct model representing the systems expected behaviour. These three challenges are also identified in the systematic literature review. Other challenges from the literature review include non-technical difficulties concerning training and the need to motivate staff and management.Using a limited case-study and a simple system three challenges, which are also identified in the literature review, throughout the MBT process are confirmed. MBT appears primarily as an academic subject with some examples of use in the industry.