Det finns en omfattande litteratur kring modellbaserad testning (MBT) men med få tecken på att metoden har fått något större genomslag i industrin. Målet med studien är att identifiera exempel där MBT används som testmetod inom industrin och eventuella hinder som finns i MBT-processen. För att genomföra detta används en mixed-methods ansats bestående av en systematisk litteraturstudie följt av en utforskande fallstudie. I fallstudien tillämpas MBT med hjälp av verktyget Modbat på ett mjukvarusystem. Endast ett fåtal industriella tillämpningar av MBT identifieras i litteraturstudien. Totalt sju studier kvarstår efter fulltextgranskningen. Studierna finns primärt inom mjukvaruindustrin och flygindustrin men innehåller även exempel från hälso- sjukvård och bilindustrin. Den utforskande fallstudien indikerar tre typer av hinder. Det första är mängden arbete med, samt bristande användarvänlighet hos verktygen. Den andra är svårigheten med att skriva ett adaptionslager som integrerar systemet med verktyget och modellen för att göra testfallen körbara. Det sista hindret är det kraftiga beroendet på att modellen utformas korrekt och stämmer med systemets tilltänkta beteende. Dessa tre hinder pekas även ut i verken från litteraturstudien. Vidare pekas bland annat även icke-tekniska svårigheter ut under litteraturstudien i form av att hela arbetsgruppen och ledningen behöver engageras för att införa ett nytt arbetssätt. Med en begränsad fallstudie och ett enkelt system bekräftas tre hinder i MBT-processen som även identifieras i litteraturgenomgången. MBT framstår som ett primärt akademiskt område med ett fåtal exempel på användning inom industrin. / There is extensive literature concerning model-based testing (MBT) but few signs that the method have had any major breakthrough in the industry. The goal of this study is to identify examples of MBT being used in the industry and any challenges faced during the MBT-process. The study is conducted using a mixed methods approach, consisting of a systematic literature review followed by an exploratory case-study. The case-study applies MBT to a software system using the MBT-tool Modbat.Only seven studies remain after the fulltext review is performed. The studies are primarily from the software and aerospace industries but also include examples from the healthcare and automotive industries. The exploratory case-study identifies three challenges. The first one is the amount of work and lacking usability related to the MBT-tools. The second challenge is implementing the adaption layer, integrating the system under test with the tool and model to make test cases executable. The final challenge is the dependency on a correct model representing the systems expected behaviour. These three challenges are also identified in the systematic literature review. Other challenges from the literature review include non-technical difficulties concerning training and the need to motivate staff and management.Using a limited case-study and a simple system three challenges, which are also identified in the literature review, throughout the MBT process are confirmed. MBT appears primarily as an academic subject with some examples of use in the industry.

Background Requirements engineering and verification (REV) processes play es-sential roles in software product development. There are physical and non-physicaldistances between entities (actors, artifacts, and activities) in these processes. Cur-rent practices that reduce the distances, such as automated testing and alignmentof document structure and tracing only partially close the above mentioned gap.Objective The aim of this thesis is to investigate solutions w.r.t their abilityto reduce the distances between requirements engineering and verification. Twotechniques that are explored in this thesis are automated testing (model-basedtesting, MBT) and alignment of document structure and tracing (traceability).Method The research methods used in this thesis are systematic mapping, soft-ware requirements mining, case study, literature survey, validation study, and de-sign science.Results MBT and traceability are effective in reducing the distance between re-quirements and verification. However, both activities have some shortcoming thatneeds to be addressed when used for that purpose. Current MBT techniques inthe context of software performance do not attain all the goals of MBT: 1) require-ments validation, 2) checking the testability of requirements, and 3) the generationof an efficient test suite. These goals are essential to reduce the distance. We de-veloped and assessed performance requirements verification and test environmentgeneration approach to tackle these shortcomings. Also, traceability between re-quirements and verification suffers from the low granularity of trace links and doesnot support the verification of all requirements. We propose the use of taxonomictrace links to trace and align the structure of requirements specifications and ver-ification artifacts. The results from the validation study show that the solution isfeasible in practice. However, this comes with challenges that need to be addressed.Conclusion MBT and improved traceability reduce multiple distances betweenactors, artifacts, and activities in the requirements engineering and verificationprocess. MBT is most effective in reducing the distances when the model used isbuilt from the requirements. Traceability is essential in easing access to relevantinformation when needed and should not be seen as an overhead. When creatingtrace links, we need to consider the difference in the abstraction, structure, andtime between the linked artifacts / <p>Chapter 3 and 4 are papers submitted to journals, and therefore removed from the fulltext file.</p>

Yes / This paper concerns the systematic testing of robotic control software based on state-based models. We focus on cyclic systems that typically receive inputs (values from sensors), perform computations, produce outputs (sent to actuators) and possibly change state. We provide a testing theory for such cyclic systems where time can be represented and probabilities are used to quantify non-deterministic choices, making it possible to model probabilistic algorithms. In addition, refusals, the inability of a system to perform a set of actions, are taken into account. We consider several possible testing scenarios. For example, a tester might only be able to passively observe a sequence of events and so cannot check probabilities, while in another scenario a tester might be able to repeatedly apply a test case and so estimate the probabilities of sequences of events. These different testing scenarios lead to a range of implementation relations (notions of correctness). As a consequence, this paper provides formal definitions of implementation relations that can form the basis of sound automated testing in a range of testing scenarios. We also validate the implementation relations by showing how observers can be used to provide an alternative but equivalent characterisation. / This work has been supported by EPSRC, United Kingdom grant EP/R025134/2 RoboTest: Systematic Model-Based Testing and Simulation of Mobile Autonomous Robots, the Spanish MINECO-FEDER grant PID2021- 122215NB-C31 (AwESOMe) and the Region of Madrid grant S2018/TCS-4314 (FORTE-CM) co-funded by EIE Funds of the European Union.

La phase la plus consommatrice de ressources d'une production de logiciels est l'étape de vérification qualité, incluant la vérification fonctionnelle des programmes. Donc, à cause de ces coûts, l'industrie de développement des logiciels exécute rarement une vérification minutieuse de ses produits. Une des stratégies les plus prometteuses pour réduire les coûts de vérification des logiciels est de rendre cette vérification aussi automatique que possible. Actuellement, cette industrie repose essentiellement sur une seule phase de tests pour vérifier les fonctionnalités des programmes. Nous avons donc cherché à automatiser le test fonctionnel des logiciels en fournissant une assistance pour l'étape de génération des scénarios de test. Le test basé sur des modèles (MBT) est une théorie de test qui a eu un grand succès dans l'automatisation du processus de test. Une méthode MBT analyse un modèle formel ou une spécification d'un logiciel pour produire des scénarios de test qui sont exécutés plus tard par le programme. La plupart de ces méthodes travaillent avec des machines à états finis (FSM) qui limite leur application puisque les FSMs ne peuvent pas décrire tous les logiciels. Cependant, un gros avantage de ces méthodes est qu'elles atteignent un haut degré d'automatisation. Dans cette thèse nous montrons, au contraire, le niveau d'automatisation que nous avons obtenu en appliquant une méthode MBT, connue comme Test Template Framework (TTF), sur des spécifications Z. Comme Z est basé sur la logique de premier ordre et la théorie du jeu, il est beaucoup plus expressif que les FSMs, rendant ainsi nos résultats applicables à une plus large gamme de programmes. / The most resource-consuming phase of software production is the verification of its qualities, including functional correctness. However, due to its costs, the software industry seldom performs a thorough verification of its products. One of the most promising strategies to reduce the costs of software verification is making it as automatic as possible. Currently, this industry relies basically only on testing to verify functional correctness. Therefore, we sought to automate functional testing of software systems by providing tool support for the test case generation step. Model-based testing (MBT) is a testing theory that has achieved impressive successes in automating the testing process. Any MBT method analyses a formal model or specification of the system under test to generate test cases that later are executed on the system. Almost all methods work with some form of finite state machines (FSM) which limits their application since FSM's cannot describe general systems. However, a great advantage of these methods is that they reach a high degree of automation. In this thesis we show, on the contrary, the degree of automation we have achieved by applying a MBT method, known as Test Template Framework (TTF), to Z specifications. Since Z is based on first-order logic and set theory it is far more expressive than FSM's, thus making our results applicable to a wider range of programs. During this thesis we have improved the TTF and developed a tool, called Fastest, that implements all of our ideas.

Software testing is a very important phase in software development as it minimize risks ina software system, however, it consumes time and can be very expensive. With automatictest case generation time consumption and cost can be reduced. Model-based testing isa method to test a software system with a model of the systems behaviour. Automatictest case generation is often considered a favorable support in model-based testing. In thiswork, the concept of model-based testing is explored along with testing the embedded partof a water-level measurement system (WLM) to investigate the efficiency of model-basedtesting on a software system. As a result of this, a model-based testing tool, MoMut::UMLis used to generate the test-cases on the UML model of WLM system that is built ina UML modeling environment, Eclipse-Papyrus. However, MoMut::UML implements aspecial type of model-based testing technique, model-based mutation testing; that injectsfaults in the UML model, and generates test-data on the fault-based model. By this, thebehaviour of system-under-test, only the UML model of water-level measurement system,is tested.

Master of Science / Department of Computing and Information Sciences / Robby / This report is a case study of applying Model-Based testing approach using SpecExplorer, which is a model-based testing tool developed by Microsoft, to test a component of Microsoft Network Monitor. The system under test is the UI of the Network Monitor feature, Parser Profiles Management. Model-Based testing is a methodology for automated testing which not only automates the test execution, but the test design and generation. This approach starts by expressing an abstract model of the system which is a smaller subset of the product behavior, but retains essential elements which form the focus of the testing problem. A model-based testing tool creates a finite state machine from the model which is traversed to produce test cases. Thus, it provides more efficient coverage and flexibility in developing and maintaining test cases.

<p>Model Based Testing (MBT) is a new-age test automation technique traditionally used for Functional Black-Box Testing. Its capability of generating test cases by using model developed from the analysis of the abstract behavior of the System under Test is gaining popularity. Many commercial and open source MBT tools are available currently in market. But each one has its own specific way of modeling and test case generation mechanism that is suitable for varied types of systems. Ericsson, a telecommunication equipment provider company, is currently adapting Model Based Testing in some of its divisions for functional testing. Those divisions haven’t yet attempted adapting Model Based Testing for non-functional testing in a full-pledged manner. A comparative study between various MBT tools will help one of the Ericsson’s testing divisions to select the best tool for adapting to its existing test environment. This also helps in improving the quality of testing while reducing cost, time and effort. This thesis work helps Ericsson testing division to select such an effective MBT tool. Based on aspects such as functionality, flexibility, adaptability, performance etc., a comparative study is carried out on various available MBT tools and a few were selected among them: Qtronic, ModelJUnit and Elvior Motes.This thesis also helps to understand the usability of the selected tools for modeling of non-functional requirements using a new method. A brief idea of modeling the non-functional requirements is suggested in this thesis. A System under Test was identified and its functional behavior was modeled along with the non functional requirements in Qtronic and ModelJUnit. An experimental analysis, backed by observations of using the new proposed method indicates that the method is efficient enough to carry out modeling non-functional requirements along with modeling of functional requirements by identifying the appropriate approach.Model Based Testing (MBT) is a new-age test automation technique traditionally used for Functional Black-Box Testing. Its capability of generating test cases by using model developed from the analysis of the abstract behavior of the System under Test is gaining popularity. Many commercial and open source MBT tools are available currently in market. But each one has its own specific way of modeling and test case generation mechanism that is suitable for varied types of systems. Ericsson, a telecommunication equipment provider company, is currently adapting Model Based Testing in some of its divisions for functional testing. Those divisions haven’t yet attempted adapting Model Based Testing for non-functional testing in a full-pledged manner. A comparative study between various MBT tools will help one of the Ericsson’s testing divisions to select the best tool for adapting to its existing test environment. This also helps in improving the quality of testing while reducing cost, time and effort. This thesis work helps Ericsson testing division to select such an effective MBT tool. Based on aspects such as functionality, flexibility, adaptability, performance etc., a comparative study is carried out on various available MBT tools and a few were selected among them: Qtronic, ModelJUnit and Elvior Motes.</p><p>This thesis also helps to understand the usability of the selected tools for modeling of non-functional requirements using a new method. A brief idea of modeling the non-functional requirements is suggested in this thesis. A System under Test was identified and its functional behavior was modeled along with the non functional requirements in Qtronic and ModelJUnit. An experimental analysis, backed by observations of using the new proposed method indicates that the method is efficient enough to carry out modeling non-functional requirements along with modeling of functional requirements by identifying the appropriate approach.</p>

<p>Testing is a critical activity in the software development process in order to obtain systems of high quality. Tieto typically develops complex systems, which are currently tested through a large number of manually designed test cases. Recent development within software testing has resulted in methods and tools that can automate the test case design, the generation of test code and the test result evaluation based on a model of the system under test. This testing approach is called model-based testing (MBT).</p><p>This thesis is a feasibility study of the model-based testing concept and has been performed at the Tieto office in Karlstad. The feasibility study included the use and evaluation of the model-based testing tool Qtronic, developed by Conformiq, which automatically designs test cases given a model of the system under test as input. The experiments for the feasibility study were based on the incremental development of a test object, which was the client protocol module of a simplified model for an ATM (Automated Teller Machine) client-server system. The experiments were evaluated both individually and by comparison with the previous experiment since they were based on incremental development. For each experiment the different tasks in the process of testing using Qtronic were analyzed to document the experience gained as well as to identify strengths and weaknesses.</p><p>The project has shown the promise inherent in using a model-based testing approach. The application of model-based testing and the project results indicate that the approach should be further evaluated since experience will be crucial if the approach is to be adopted within Tieto’s organization.</p>

This dissertation concerns testability of event-triggered real-time systems. Real-time systems are known to be hard to test because they are required to function correct both with respect to what the system does and when it does it. An event-triggered real-time system is directly controlled by the events that occur in the environment, as opposed to a time-triggered system, which behavior with respect to when the system does something is constrained, and therefore more predictable. The focus in this dissertation is the behavior in the time domain and it is shown how testability is affected by some factors when the system is tested for timeliness. This dissertation presents a survey of research that focuses on software testability and testability of real-time systems. The survey motivates both the view of testability taken in this dissertation and the metric that is chosen to measure testability in an experiment. We define a method to generate sets of traces from a model by using a meta algorithm on top of a model checker. Defining such a method is a necessary step to perform the experiment. However, the trace sets generated by this method can also be used by test strategies that are based on orderings, for example execution orders. An experimental study is presented in detail. The experiment investigates how testability of an event-triggered real-time system is affected by some constraining properties of the execution environment. The experiment investigates the effect on testability from three different constraints regarding preemptions, observations and process instances. All of these constraints were claimed in previous work to be significant factors for the level of testability. Our results support the claim for the first two of the constraints while the third constraint shows no impact on the level of testability. Finally, this dissertation discusses the effect on the event-triggered semantics when the constraints are applied on the execution environment. The result from this discussion is that the first two constraints do not change the semantics while the third one does. This result indicates that a constraint on the number of process instances might be less useful for some event-triggered real-time systems.

Model Based Testing (MBT) is a new-age test automation technique traditionally used for Functional Black-Box Testing. Its capability of generating test cases by using model developed from the analysis of the abstract behavior of the System under Test is gaining popularity. Many commercial and open source MBT tools are available currently in market. But each one has its own specific way of modeling and test case generation mechanism that is suitable for varied types of systems. Ericsson, a telecommunication equipment provider company, is currently adapting Model Based Testing in some of its divisions for functional testing. Those divisions haven’t yet attempted adapting Model Based Testing for non-functional testing in a full-pledged manner. A comparative study between various MBT tools will help one of the Ericsson’s testing divisions to select the best tool for adapting to its existing test environment. This also helps in improving the quality of testing while reducing cost, time and effort. This thesis work helps Ericsson testing division to select such an effective MBT tool. Based on aspects such as functionality, flexibility, adaptability, performance etc., a comparative study is carried out on various available MBT tools and a few were selected among them: Qtronic, ModelJUnit and Elvior Motes.This thesis also helps to understand the usability of the selected tools for modeling of non-functional requirements using a new method. A brief idea of modeling the non-functional requirements is suggested in this thesis. A System under Test was identified and its functional behavior was modeled along with the non functional requirements in Qtronic and ModelJUnit. An experimental analysis, backed by observations of using the new proposed method indicates that the method is efficient enough to carry out modeling non-functional requirements along with modeling of functional requirements by identifying the appropriate approach.Model Based Testing (MBT) is a new-age test automation technique traditionally used for Functional Black-Box Testing. Its capability of generating test cases by using model developed from the analysis of the abstract behavior of the System under Test is gaining popularity. Many commercial and open source MBT tools are available currently in market. But each one has its own specific way of modeling and test case generation mechanism that is suitable for varied types of systems. Ericsson, a telecommunication equipment provider company, is currently adapting Model Based Testing in some of its divisions for functional testing. Those divisions haven’t yet attempted adapting Model Based Testing for non-functional testing in a full-pledged manner. A comparative study between various MBT tools will help one of the Ericsson’s testing divisions to select the best tool for adapting to its existing test environment. This also helps in improving the quality of testing while reducing cost, time and effort. This thesis work helps Ericsson testing division to select such an effective MBT tool. Based on aspects such as functionality, flexibility, adaptability, performance etc., a comparative study is carried out on various available MBT tools and a few were selected among them: Qtronic, ModelJUnit and Elvior Motes. This thesis also helps to understand the usability of the selected tools for modeling of non-functional requirements using a new method. A brief idea of modeling the non-functional requirements is suggested in this thesis. A System under Test was identified and its functional behavior was modeled along with the non functional requirements in Qtronic and ModelJUnit. An experimental analysis, backed by observations of using the new proposed method indicates that the method is efficient enough to carry out modeling non-functional requirements along with modeling of functional requirements by identifying the appropriate approach.