A modern truck contains a large number of functionalities implemented in its electronics system. Thus, testing all of these functions employs a considerable effort. The execution of tests against the system has been automated for a long time. Unfortunately, most of the test is still designed manually these days. This manual test design is sometimes not comprehensive enough to cover all possible scenarios within a complex system. At the moment, there is also a growing trend in the development process based on the model. Furthermore, model-based software can handle events and signal behaviour more robustly[1]. This thesis investigates the technique in model-based testing. The study evaluates the requirement modelling and automated abstract test generation of model-based testing over the existing testing method. A cause-effect graph is utilized for the modelling in Matlab Simulink tool with DesignVerifier feature. The case study is the truck dosing system in Scania. The results are the following. The temporal and static requirements modelling are capable of being modelled using the cause-effect graph in Matlab Simulink. Compared to the traditional method, the MBT method can achieve higher requirement coverage and more rigorous test with optimized test case generation. The MBT method also has a rapid test case generation time suitable for quick design iteration. However, the total test development time (including test case generation time) of using MBT is 12.5% higher than the manual method. Using a model-based platform like Matlab Simulink is recommended to assist the manual testing, not to replace the test flow entirely with the current research state. / En modern truck innehåller ett stort antal funktioner implementerade i dess elektroniksystem. Att testa alla dessa funktioner kräver därför en avsevärd ansträngning. Utförandet av tester mot systemet har varit automatiserat under lång tid. Tyvärr är det mesta av testet fortfarande utformat manuellt nu för tiden. Denna manuella testdesign är ibland inte tillräckligt omfattande för att täcka alla möjliga scenarier inom ett komplext system. För tillfället finns det också en växande trend i utvecklingsprocessen utifrån modellen. Dessutom kan modellbaserad programvara hantera händelser och signalbeteende mer robust[1]. Detta examensarbete undersöker tekniken i modellbaserad testning. Studien utvärderar kravmodellering och automatiserad abstrakt testgenerering av modellbaserad testning över den befintliga testmetoden. En cause-effect graph används för modelleringen i Matlab Simulink-verktyget med Design Verifier-funktionen. Fallstudien är lastbilens doseringssystem i Scania. Resultaten är följande. Den tidsmässiga och statiska kravmodelleringen kan modelleras med hjälp av cause-effect graph i Matlab Simulink. Jämfört med den traditionella metoden kan MBT-metoden uppnå högre kravtäckning och mer rigorösa test med optimerad testfallsgenerering. MBT-metoden har också en snabb genereringstid för testfall som är lämplig för snabb designiteration. Den totala testutvecklingstiden (inklusive genereringstid för testfall) för att använda MBT är 12,5% högre än den manuella metoden. Att använda en modellbaserad plattform som Matlab Simulink rekommenderas för att underlätta den manuella testningen, inte för att ersätta testflödet helt med det aktuella forskningsläget.

No description available.

Submitted by Natalia de Souza Gonçalves (natalia.goncalves@ufpe.br) on 2016-09-28T12:33:15Z No. of bitstreams: 2 license_rdf: 1232 bytes, checksum: 66e71c371cc565284e70f40736c94386 (MD5) GustavoHPCarvalho_Doutorado_CInUFPE_2016.pdf: 1763137 bytes, checksum: aed7b3ab2f6235757818003678633c9b (MD5) / Made available in DSpace on 2016-09-28T12:33:15Z (GMT). No. of bitstreams: 2 license_rdf: 1232 bytes, checksum: 66e71c371cc565284e70f40736c94386 (MD5) GustavoHPCarvalho_Doutorado_CInUFPE_2016.pdf: 1763137 bytes, checksum: aed7b3ab2f6235757818003678633c9b (MD5) Previous issue date: 2016-02-26 / High trustworthiness levels are usually required when developing critical systems, and model based testing (MBT) techniques play an important role generating test cases from specification models. Concerning critical systems, these models are usually created using formal or semi-formal notations. Moreover, it is also desired to clearly and formally state the conditions necessary to guarantee that an implementation is correct with respect to its specification by means of a conformance relation, which can be used to prove that the test generation strategy is sound. Despite the benefits of MBT, those who are not familiar with the models syntax and semantics may be reluctant to adopt these formalisms. Furthermore, most of these models are not available in the very beginning of the project, when usually natural-language requirements are available. Therefore, the use of MBT is postponed. Here, we propose an MBT strategy for generating test cases from controlled naturallanguage (CNL) requirements: NAT2TEST, which refrains the user from knowing the syntax and semantics of the underlying notations, besides allowing early use of MBT via naturallanguage processing techniques; the formal and semi-formal models internally used by our strategy are automatically generated from the natural-language requirements. Our approach is tailored to data-flow reactive systems: a class of embedded systems whose inputs and outputs are always available as signals. These systems can also have timed-based behaviour, which may be discrete or continuous. The NAT2TEST strategy comprises a number of phases. Initially, the requirements are syntactically analysed according to a CNL we proposed to describe data-flow reactive systems. Then, the requirements informal semantics are characterised based on the case grammar theory. Afterwards, we derive a formal representation of the requirements considering a model of dataflow reactive systems we defined. Finally, this formal model is translated into communicating sequential processes (CSP) to provide means for generating test cases. We prove that our test generation strategy is sound with respect to our timed input-output conformance relation based on CSP: csptio. Besides CSP, we explore the generation of other target notations (SCR and IMR) from which we can generate test cases using commercial tools (T-VEC and RT-Tester, respectively). The whole process is fully automated by the NAT2TEST tool. Our strategy was evaluated considering examples from the literature, the aerospace (Embraer) and the automotive (Mercedes) industry. We analysed performance and the ability to detect defects generated via mutation. In general, our strategy outperformed the considered baseline: random testing. We also compared our strategy with relevant commercial tools. / Testes baseados em modelos (MBT) consiste em criar modelos para especificar o comportamento esperado de sistemas e, a partir destes, gerar testes que verificam se implementações possuem o nível de confiabilidade esperado. No contexto de sistemas críticos, estes modelos são normalmente (semi)formais e deseja-se uma definição precisa das condições necessárias para garantir que uma implementação é correta em relação ao modelo da especificação. Esta definição caracteriza uma relação de conformidade, que pode ser usada para provar que uma estratégia de MBT é consistente (sound). Apesar dos benefícios, aqueles sem familiaridade com a sintaxe e a semântica dos modelos empregados podem relutar em adotar estes formalismos. Aqui, propõe-se uma estratégia de MBT para gerar casos de teste a partir de linguagem natural controlada (CNL). Esta estratégia (NAT2TEST) dispensa a necessidade de conhecer a sintaxe e a semântica das notações formais utilizadas internamente, uma vez que os modelos intermediários são gerados automaticamente a partir de requisitos em linguagem natural. Esta estratégia é apropriada para sistemas reativos baseados em fluxos de dados: uma classe de sistemas embarcados cujas entradas e saídas estão sempre disponíveis como sinais. Estes sistemas também podem ter comportamento dependente do tempo (discreto ou contínuo). Na estratégia NAT2TEST, inicialmente, os requisitos são analisados sintaticamente de acordo com a CNL proposta neste trabalho para descrever sistemas reativos. Em seguida, a semântica informal dos requisitos é caracterizada utilizando a teoria de gramática de casos. Posteriormente, deriva-se uma representação formal dos requisitos considerando um modelo definido neste trabalho para sistemas reativos. Finalmente, este modelo é traduzido em uma especificação em communicating sequential processes (CSP) para permitir a geração de testes. Este trabalho prova que a estratégia de testes proposta é consistente considerando a relação de conformidade temporal baseada em entradas e saídas também definida aqui: csptio. Além de CSP, foi explorada a geração de outras notações formais (SCR e IMR), a partir das quais é possível gerar casos de teste usando ferramentas comerciais (T-VEC e RT-Tester, respectivamente). Todo o processo é automatizado pela ferramenta NAT2TEST. A estratégia NAT2TEST foi avaliada considerando exemplos da literatura, da indústria aeroespacial (Embraer) e da automotiva (Mercedes). Foram analisados o desempenho e a capacidade de detectar defeitos gerados através de operadores de mutação. Em geral, a estratégia NAT2TEST apresentou melhores resultados do que a referência adotada: testes aleatórios. A estratégia NAT2TEST também foi comparada com ferramentas comerciais relevantes.

Cette thèse propose une approche originale de test de vulnérabilité Web à partir de modèles etdirigée par des patterns de tests, nommée PMVT. Son objectif est d’améliorer la capacité de détectionde quatre types de vulnérabilité majeurs, Cross-Site Scripting, Injections SQL, Cross-Site RequestForgery, et Privilege Escalation. PMVT repose sur l’utilisation d’un modèle comportemental del’application Web, capturant ses aspects fonctionnels, et sur un ensemble de patterns de test devulnérabilité qui adressent un type de vulnérabilité de manière générique, quelque soit le type del’application Web sous test.Par l’adaptation de technologies MBT existantes, nous avons développé une chaîne outillée complèteautomatisant la détection des quatre types de vulnérabilité. Ce prototype a été exprimenté et évaluésur deux applications réelles, actuellement utiliseés par plusieurs dizaines de milliers d’utilisateurs.Les résultats d’expérimentation démontrent la pertinence et de l’efficience de PMVT, notamment enaméliorant de façon significative la capacité de détection de vulnérabilités vis à vis des scannersautomatiques d’applications Web existants. / This thesis proposes an original approach, dubbed PMVT for Pattern-driven and Model-basedVulnerability Testing, which aims to improve the capability for detecting four high-profile vulnerabilitytypes, Cross-Site Scripting, SQL Injections, CSRF and Privilege Escalations, and reduce falsepositives and false negatives verdicts. PMVT relies on the use of a behavioral model of theapplication, capturing its functional aspects, and a set of vulnerability test patterns that addressvulnerabilities in a generic way. By adapting existing MBT technologies, an integrated toolchain that supports PMVT automates thedetection of the four vulnerability types in Web applications. This prototype has been experimentedand evaluated on two real-life Web applications that are currently used by tens of thousandsusers. Experiments have highlighted the effectiveness and efficiency of PMVT and shown astrong improvement of vulnerability detection capabilities w.r.t. available automated Web applicationscanners for these kind of vulnerabilities.

Cieľom tejto práce je vývoj nástroja na automatické generovanie testov pre aplikácie s grafickým užívateľským rozhraním v~prostredí GNOME. Na generovanie testov sú použité metadáta asistenčných technológií, konrétne AT-SPI. Navrhnutý generátor testov využíva dané metadáta na vytvorenie modelu testovanej aplikácie. Model mapuje sekvencie udalostí, ktoré generátor vykoná na testovanej aplikácii počas generovania testov. Súčasťou procesu generovania je zároveň detekcia závažných chýb v testovaných aplikáciách. Výstupom procesu generovania sú automatizované testy, ktoré sú vhodné na regresné testovanie. Funkčnosť implementovaného generátora testov bola úspešne overená testovaním 5 aplikácií s otvoreným zdrojovým kódom. Počas testovania aplikácií navrhnutým nástrojom sa preukázala schopnosť detekovať nové chyby.

Le test à base de modèles des systèmes réactifs est le processus de vérifier si un système sous test (SUT) est conforme à sa spécification. Il consiste à gérer à la fois la génération des données de test et le calcul de verdicts en utilisant des modèles. Nous spécifions le comportement des systèmes réactifs à l'aide des systèmes de transitions symboliques temporisées à entrée-sortie (TIOSTS). Quand les TIOSTSs sont utilisés pour tester des systèmes avec une interface centralisée, l'utilisateur peut ordonner complètement les événements (i.e., les entrées envoyées au système et les sorties produites). Les interactions entre le testeur et le SUT consistent en des séquences d'entrées et de sortie nommées traces, pouvant être séparées par des durées dans le cadre du test temporisé, pour former ce que l'on appelle des traces temporisées. Les systèmes distribués sont des collections de composants locaux communiquant entre eux et interagissant avec leur environnement via des interfaces physiquement distribuées. Différents événements survenant à ces différentes interfaces ne peuvent plus être ordonnés. Cette thèse concerne le test de conformité des systèmes distribués où un testeur est placé à chaque interface localisée et peut observer ce qui se passe à cette interface. Nous supposons qu'il n'y a pas d’horloge commune mais seulement des horloges locales pour chaque interface. La sémantique de tels systèmes est définie comme des tuples de traces temporisées. Nous considérons une approche du test dans le contexte de la relation de conformité distribuée dtioco. La conformité globale peut être testée dans une architecture de test en utilisant des testeurs locaux sans communication entre eux. Nous proposons un algorithme pour vérifier la communication pour un tuple de traces temporisées en formulant le problème de message-passing en un problème de satisfaction de contraintes (CSP). Nous avons mis en œuvre le calcul des verdicts de test en orchestrant à la fois les algorithmes du test centralisé off-line de chacun des composants et la vérification des communications par le biais d'un solveur de contraintes. Nous avons validé notre approche sur un cas étude de taille significative. / Model-based testing of reactive systems is the process of checking if a System Under Test (SUT) conforms to its model. It consists of handling both test data generation and verdict computation by using models. We specify the behaviour of reactive systems using Timed Input Output Symbolic Transition Systems (TIOSTS) that are timed automata enriched with symbolic mechanisms to handle data. When TIOSTSs are used to test systems with a centralized interface, the user may completely order events occurring at this interface (i.e., inputs sent to the system and outputs produced from it). Interactions between the tester and the SUT are sequences of inputs and outputs named traces, separated by delays in the timed framework, to form so-called timed traces. Distributed systems are collections of communicating local components which interact with their environment at physically distributed interfaces. Interacting with such a distributed system requires exchanging values with it by means of several interfaces in the same testing process. Different events occurring at different interfaces cannot be ordered any more. This thesis focuses on conformance testing for distributed systems where a separate tester is placed at each localized interface and may only observe what happens at this interface. We assume that there is no global clock but only local clocks for each localized interface. The semantics of such systems can be seen as tuples of timed traces. We consider a framework for distributed testing from TIOSTS along with corresponding test hypotheses and a distributed conformance relation called dtioco. Global conformance can be tested in a distributed testing architecture using only local testers without any communication between them. We propose an algorithm to check communication policy for a tuple of timed traces by formulating the verification of message passing in terms of Constraint Satisfaction Problem (CSP). Hence, we were able to implement the computation of test verdicts by orchestrating both localised off-line testing algorithms and the verification of constraints defined by message passing that can be supported by a constraint solver. Lastly, we validated our approach on a real case study of a telecommunications distributed system.

Model-based testing (MBT) is a black-box software testing technique that focuses on specification of the system-under-test (SUT) and/or its environment. It uses models to automatically generate a large number of tests. To the best of our knowledge, no study has investigated the correlation of model coverage with SUT coverage using more advanced coverage criteria (such as edge-pair coverage) and the correlation of coverage (at model level and SUT level) with test suite effectiveness using non-adequate test suites in the context of MBT despite the prominence of non-adequate test suites in industry. To carry out the investigation, we extend an existing open-source MBT tool called Modbat to measure edge-pair coverage at model level, implement a new tool called PaCovForJbc to measure edge-pair coverage, edge coverage and node coverage at SUT level. Finally, we perform an experiment using these tools applied on three projects: “ArrayList”, and “LinkedList” of Java standard library, and “Apache ZooKeeper”. Overall, the results suggest the following: Edge and edge-pair coverage at model level often have a moderate to high correlation with the same type of coverage at SUT level, while that link between model and SUT for node coverage is weaker. Moreover, coverage criteria at SUT level often have a moderate to high correlation with test suite effectiveness, and a coverage criterion at SUT level has a slightly higher correlation with test suite effectiveness than the same type of coverage at model level. Regarding coverage at model level, edge and edge-pair coverage at model level have a slightly higher correlation with test suite effectiveness than node coverage at model level. Note that the mentioned suggestions need to be taken with discretion, because results vary depending on the project and/or coverage criterion under investigation. / Modellbaserad testning (MBT) är en black-box-testteknik som fokuserar på specifikation av system-under-test (SUT) och/eller dess miljö. MBT använder modeller för att generera ett stort antal tester automatiskt. Såvitt vi vet, finns ingen studie som undersökt korrelationen mellan modelltäckning och täckning av SUT med hjälp av mer avancerade täckningskriterier såsom kantparstäckning. Dessutom finns ingen studie som undersökt korrelationen mellan täckning (på modellnivå och SUT-nivå) och effektivitet av icke- adekvata testsviter som genereras med hjälp av MBT trots betydelsen av icke-adekvata testsviter i industrin. För att utföra undersökningen, utökar vi ett ”open-source” MBT-verktyg som kallas för Modbat för att mäta kantparstäckning på modellnivå. Dessutom implementerar vi ett nytt verktyg som kallas för PaCovForJbc för att mäta kantpars-, kant- och nodtäckning på SUT-nivå. Till slut utför vi experiment genom att applicera Modbat och PaCovForJbc på tre projekt: ”ArrayList” och ”LinkedList” av Javas standardbibliotek samt ”Apache ZooKeeper”. Sammantaget indikerar resultaten följande: Kant- och kantparstäckning på modellnivå har ofta en måttlig till hög korrelation med samma typ av täckning på SUT- nivå, medan länken mellan modell och SUT för nodtäckning är svagare. Dessutom har täckningskriterier på SUT-nivå ofta en måttlig till hög korrelation med testsvitseffektivitet, och ett täckningskriterium på SUT-nivå har en aning högre korrelation med testsvitseffektivitet än samma typ av täckning på modellnivå. Angående täckning på modellnivå har kant- och kantparstäckning på modellnivå en aning högre korrelation med testsvitseffektivitet än nodtäckning på modellnivå. Observera att de nämnda förslagen måste tas med diskretion, eftersom resultaten varierar beroende på projektet och/eller täckningskriteriet som undersöks.