Est-il possible de tester automatiquement le comportement temporisé des systèmes interactifs temps réel ? Ces travaux proposent une solution en fournissant un ensemble d’outils de test basé sur modèles pour Systèmes Musicaux Interactifs (SMI). Les SMIs doivent calculer et réagir pendant une performance musicale et ainsi accompagner les musiciens. Certains de ces SMIs peuvent être basés sur partition et doivent, dans ce cas, suivre à tout prix les contraintes temporelles imposées par le document haut-niveau appelé partition. En somme, pendant une performance, le système doit réagir en temps réel aux signaux audio venant des musiciens en suivant cette partition. Ceci demande au système une forte fiabilité temporelle et une robustesse face aux erreurs pouvant arriver en entrée du système. Hors, la vérification formelle de propriétés, comme la fiabilité temporelle avant l’exécution du système lors d’une performance, est insuffisamment traitée par la communauté de l’informatique musicale. Nous présentons dans cette thèse, la réalisation d’un ensemble d’outils de test basé sur modèles appliqué à un SMI. Il est à noter que ces outils de test ont été définis formellement dans le but de tester plus généralement le comportement temporelle des systèmes interactifs temps réel prenant en compte des évènements discrets et des durées définissables sur des échelles multiples. Pour ce résumé nous présentons rapidement l’état de l’art de nos travaux avant d’introduire la définition de notre modèle créé pour spécifier les aspects évènementiel («event-triggerred») et temporel («timed-driven») des SMIs. Ce modèle a la particularité d’être automatiquement construit depuis les conditions temporelles définies dans un document haut-niveau et peut être traduit vers un réseau d’Automates Temporisés (TA). Dans le cadre de la performance musique mixte électronique/instrumentale nous avons introduit une notion de durée multi-temps gérée par notre modèle et une génération de trace d’entrée musicalement pertinente par notre ensemble d’outils de test. Pour tester un SMI selon les différentes attentes de l’utilisateur, notre ensemble d’outils a été implémenté avec plusieurs options possibles. Parmi ces options, la possibilité de tester automatiquement, selon une approche différée ou temps réel, la conformité temporelle du SMI est proposée. En effet, l’approche différée utilise des outils de la gamme du logiciel Uppaal [44] pour générer une suite de traces d’entrées exhaustive et garantir la conformité temporelle du système testé. Il est également possible de tester une trace d’entrée particulière ou une version altérée («fuzzed») de la trace idéale définie par la partition. L’approche temps réel interprète quand-à elle directement le modèle comme des instructions de byte-code grâce à une machine virtuelle. Finalement, des expériences ont été conduites via une étude de cas sur le suiveur de partition Antescofo. Ces expériences ont permis de tester ce système et d’évaluer notre ensemble d’outils et ses différentes options. Ce cas d’étude applique nos outils de test sur Antescofo avec succès et a permit d’identifier des bogues parfois non triviaux dans ce SMI. / Can real-time interactive systems be automatically timed tested ? This work proposes an answer to this question by providing a formal model based testing framework for Interactive Music Systems (IMS). IMSs should musically perform computations during live performances, accompanying and acting like real musicians. They can be score-based, and in this case must follow at all cost the timed high-level requirement given beforehand, called score. During performance, the system must react in real-time to audio signals from musicians according to this score. Such goals imply strong needs of temporal reliability and robustness to unforeseen errors in input. Be able to formally check this robustness before execution is a problem insufficiently addressed by the computer music community. We present, in this document, the concrete application of a Model-Based Testing (MBT) framework to a state-of-the-art IMS. The framework was defined on purpose of testing real-time interactive systems in general. We formally define the model in which our method is based. This model is automatically constructed from the high-level requirements and can be translated into a network of time automata. The mixed music environment implies the management of a multi-timed context and the generation of musically relevant input data through the testing framework. Therefore, this framework is both time-based, permitting durations related to different time units, and event-driven, following the musician events given in input. In order to test the IMS against the user’s requirements, multiple options are provided by our framework. Among these options, two approaches, offline and online, are possible to assess the system timed conformance fully automatically, from the requirement to the verdict. The offline approach, using the model-checker Uppaal, can generate a covering input suite and guarantee the system time reliability, or only check its behavior for a specific or fuzzed input sequence. The online approach, directly interprets the model as byte-code instructions thanks to a virtual machine. Finally, we perform experiments on a real-case study: the score follower Antescofo. These experiments test the system with a benchmark of scores and a real mixed-score given as input requirements in our framework. The results permit to compare the different options and scenarios in order to evaluate the framework. The application of our fully automatic framework to real mixed scores used in concerts have permitted to identify bugs in the target IMS.

In order to generate substantial market impact, mobile applications must be able to run on multiple platforms. Hence, software engineers face a multitude of technologies and system versions resulting in static variability. Furthermore, due to the dependence on sensors and connectivity, mobile software has to adapt its behavior accordingly at runtime resulting in dynamic variability. However, software engineers need to assure quality of a mobile application even with this large amount of variability—in our approach by the use of model-based testing (i.e., the generation of test cases from models). Recent concepts of test metamodels cannot efficiently handle dynamic variability. To overcome this problem, we propose a process for creating black-box test models based on dynamic feature Petri nets, which allow the description of configuration-dependent behavior and reconfiguration. We use feature models to define variability in the system under test. Furthermore, we illustrate our approach by introducing an example translator application.

Model-based testing (MBT) is a promising testing method with advantages like exhaustive exploration and high maintainability. However, one notable downside is that the generated tests usually contain much unnecessary noise. This noise can present itself as superfluous actions that bear no effect on test outcome — worsening comprehensibility and inflating test size. Generalpurpose minimization techniques like delta debugging have been successful in minimizing similar input before. The process involves removing elements that are redundant for satisfying given criteria, e.g., that a test still identifies a specific fault. In this thesis, we formulate the modmin algorithm which makes use of a hierarchical delta debugging approach to minimize sequences generated with Modbat — an open source MBT tool based on the extended finite-state machine (EFSM). One after the other, the algorithm attacks three common sub-structures found within the generated tests: model instances, loops, and transitions. To evaluate the work, we extended Modbat with modmin and applied it to tests generated from a set of ten models of varying complexity. The results show that modmin is very proficient at minimizing the tests generated from our model set and that it does so at a negligible cost. / Modellbaserad testning är en lovande teknik med fördelar som uttömande sökning och hög underhållbarhet. En nackdel är däremot att de genererade testfallen tenderar att innehålla onödig information. Ett testfall ska, med fördel, vara så kort och koncist som möjligt, och överflödiga instruktioner förvärrar både testbegriplighet och teststorlek. Minimeringsstrategier som delta debugging har med goda resultat används för att minimera liknande datastrukturer tidigare. Processen innebär vanligtvis att man plockar bort element som inte är nödvändiga för att särskilda kriterier ska vara uppfyllda. Exempelvis att ett test fortfarande identifierar samma fel som innan. I det här verket formulerar vi och implementerar modmin-algoritmen, en algoritm som bygger på hierarkisk delta debugging för att minimera testfall generade med det modellbaserade testningsverktyget Modbat. En efter en attackerar vår algoritm tre vanliga delstrukturer som vi har identifierat i Modbats testfall: modellinstanser, slingor, och individuella övergångar. Vi utvärderade arbetet genom att utöka Modbats öppna källkod med modmin och sen minimera testfall genererade från tio olika modeller av varierande komplexitet. Resultaten visar att modmin klarar av att minimera testfall generade från alla våra modeller och att det bara tillför en försumbar kostnad vad gäller systemresurser eller körtid.

Microarchitecture’s optimizations have increased the performance but lowered the security. Speculative execution is one of the optimizations that was thought to be secure, but it is exploitable to leak information. The problem with these exploits is that there is no easy software defence and many exploits could be unexplored due to it being a fairly recent discovery. This thesis explores a way to find code that is vulnerable to this. The solution to the problem is to use the tool Side Channel Abstract Model Validator (SCAMV) which implements the method Model-Based Testing (MBT). We examine the core CVA6, which is a RISCV Central Processing Unit (CPU). Test cases are generated by program generators and interesting ones are selected by applying an observational model to them. The observational model abstracts side-channel leakage of the microarchitecture. The selected test cases are executed on the platform to validate the used observational models. The results of the test cases showed no indication of modifying the side channels under speculative execution. The results showed that SCAMV can examine timing-based channels. The conclusion is that our findings indicate that the CVA6 core is not vulnerable to speculative cache or timing-based side-channel attacks. / Optimeringar på mikroarkitektur nivåer har ökat prestandan men minskat säkerheten. Spekulativt utförande (speculative execution) är en av de optimeringar som har ansetts vara säkert, men det har visats att det kan utnyttjas för att läcka information. Problemet med dessa sårbarheter är att det inte finns något enkelt mjukvaruförsvar och att många sårbarheter fortfarande kan vara outforskade. Denna avhandling undersöker ett sätt att försöka hitta kod som är sårbar för detta. Lösningen på problemet är att använda verktyget SCAMV som använder sig av metoden Model-Baserad Testning. Vi undersöker CVA6, vilket är en RISCV CPU. Testfall genereras av programgeneratorer och intressanta testfall väljs genom att tillämpa en observationsmodell på dem. Observationsmodellen abstraherar sidokanalläckage i mikroarkitekturen. De valda testprogrammen verkställs på plattformen för att validera de använda observationsmodellerna. Resultatet från testfallen visade ingen indikation på att det går att modifiera sidokanalerna under spekulativt utförande. Resultatet visade att SCAMV kan undersöka tidsbaserade kanaler. Slutsatsen är att våra resultat indikerar att CVA6 inte är sårbar för spekulativa cache eller tidsbaserade sidokanalattacker.

There are some systems (or parts of systems) that are very critical and need especially good test suites to test them. For these critical systems, exhaustive testing may be a good way to test them. Thus, we have implemented two versions of bounded exhaustive search (BES) algorithms in a model-based testing tool called, Modbat. One of the BES versions (BESnL) visits each self-loop in the model only once. The other version (BESL) has no constraint or limitation on the number of time it visits each self-loop. We have then compared the two BES algorithms with each other and with an already implemented algorithm in Modbat called random search (RS). We have run the three mentioned algorithms (BESL, BESnL and RS) on five different models and compared their performance on these models in terms of time, coverage and finding faults. We have found that BESnL is faster than BESL and it can miss some faults that BESL can find. However, BESnL can find errors faster than BESL. BESL has sometimes better performance than BESnL in terms of branch coverage. In terms of other coverage criteria (like state coverage, transition coverage and instruction coverage), both BESL and BESnL has very similar performance. We have also found that running the RS algorithm is, in general, faster than both BES algorithms at generating tests (given the same total number of tests generated) if the model has a clear end state. RS may also be faster at finding faults than the BES algorithms. However, The BES algorithms and the RS algorithm have similar behaviours regarding coverage. Nevertheless, RS can sometimes reach higher coverage faster than the BES algorithms and with a smaller number of tests. / Det finns vissa system (eller delar av system) som är mycket essentiella och som behöver särskilt bra testsviter för att testa dem. För dessa essentiella system kan uttömmande tester vara ett bra sätt att testa dem. Således har vi implementerat två versioner av begränsad uttömmande sökning eller på engelska ”bounded exhuastive search” (BES) algoritmer i ett modellbaserat testverktyg kallat Modbat. En av BES-versionerna (BESnL) besöker varje självslinga i modellen bara en gång. Den andra versionen (BESL) har ingen begränsning av hur många gånger den besöker varje självslinga. Vi har sedan jämfört de två BES-algoritmerna med varandra och med en redan implementerad algoritm i Modbat som kallas slumpmässig sökning eller på engelska ”random search” (RS). Vi har kört de tre nämnda algoritmerna (BESL, BESnL och RS) på fem olika modeller och jämfört deras prestanda på dessa modeller när det gäller tid, täckning (coverage) och att hitta fel. Vi har funnit att BESnL är snabbare än BESL och det kan missa några fel som BESL kan hitta, men BESnL kan hitta fel snabbare än BESL. BESL har ibland bättre prestanda än BESnL när det gäller filialtäckning (branch-coverage). När det gäller andra täckningskriterier (som statlig täckning, övergångstäckning (tranintion-coverage) och instruktionstäckning) har både BESL och BESnL mycket liknande resultat. Vi har också funnit att körning av RS-algoritmen i allmänhet är snabbare än båda BES- algoritmerna vid generering av tester (givet samma totala antal genererade tester) om modellen har ett klart slutläge (end-state). RS kan också vara snabbare att hitta fel än BES-algoritmerna. BES-algoritmerna och RS-algoritmen har dock liknande beteenden när det gäller täckning. RS kan ibland nå högre täckning snabbare än BES-algoritmerna och med ett mindre antal tester.

[pt] O uso de sistemas baseados em agentes é adequado à construção de software complexo. Para garantir uma ordem social desejável é preciso lidar com a autonomia e a diversidade de interesses dos agentes e as normas são mecanismos eficientes de controle usados para regular o comportamento dos agentes. O teste de software continua sendo amplamente aplicado para garantir a qualidade do software. No contexto de sistemas multiagentes normativos, os testes devem lidar com a necessidade dos agentes atuarem de forma robusta sob condições normativas dinâmicas as quais os desenvolvedores não consideraram. Neste contexto, propomos uma abordagem para testar agentes normativos que seguem o modelo belief-desire-intention. Como contribuições, esta tese apresenta: um framework para desenvolvimento de agentes BDI normativos, o NBDI4JADE; um modelo modelo de faltas para apoiar a identificação dos diferentes tipos de falhas em agentes normativos; um framework para testar agentes BDI normativos, o N-JAT4BDI e, um método para geração de casos de teste a partir de modelos ANA-ML dos agentes. Avaliamos o framework de teste através de um estudo experimental no qual discutimos a eficiência e a eficácia do framework. Avaliamos, também, a eficácia do método de geração de casos de teste,aplicando-o a dois cenários de uso: (i) um sistema para gerenciar a submissão e o processo de revisão de artigos em conferências, e (ii) um sistema de venda de pacotes turísticos de uma agência de viagens. Os resultados obtidos nas avaliações de ambos os frameworks apresentam indícios positivos da eficiência e eficácia na detecção e identificação de falhas em agentes normativos e eficiência na geração dos casos de teste. / [en] The use of agent-based systems is suitable for the construction of complex software. To guarantee a desirable social order one we must deal with the autonomy and diversity of interests of the agents and norms are efficient mechanisms of control used to regulate the behavior of the agents. Software testing still have been widely applied to ensure the software quality. In the context of normative multiagent systems, the test must deal with the need for the agents to act robustly under dynamic normative conditions in which the developers did not consider. In this context, we propose an approach to test normative agents that follow the belief-desire-intention model. Among the contributions, this thesis presents: a framework for development of normative BDI agents, the NBDI4JADE; a fault model to support the identification of different types of failures in normative agents; a framework for testing normative BDI agents, the N-JAT4BDI, and a method for generating test cases from ANA-ML models.We evaluated the test framework through an experimental study in which we discussed the efficiency and effectiveness of the framework. We also evaluate the effectiveness of the test case generation method, applying it to two usage scenarios: (i) a system to the manage submission and the review process of articles in conferences, and (ii) a system of sale of tour packages from a travel agency. The results obtained in the evaluations of both frameworks presented positive indications of efficiency and effectiveness in the detection and identification of failures in normative agents and efficiency in the generation of test cases.

Model-Based Testing has shown tangible benefits in previous research exploring widespread domains. Clearing systems in financial markets however are large software systems in which these approaches rarely have been explored. This paper in particular explores the Model-Based Testing approach using Modbat, a specialized tool based on extended finite-state machines, as a complementary testing approach on software clearing systems at Nasdaq. In particular, the approach is applied to functionalities of the system to test both its ability to find bugs undiscovered by traditional methods, as well as to measure its effectiveness and efficiency in creating reliable coverage in short amounts of time. In this study, we show that the model-based Modbat approach can achieve high code coverage with relatively little effort. 77 % of the production coverage was achieved with very few Modbat test runs. Additionally, only ∼ 9-14 % of the same lines of code, and a fraction of the time were required to achieve these results. From this, we conclude that Model-Based Testing approaches such as Modbat allow for significantly more efficient testing in terms of coverage over time compared to manual approaches. / Modellbaserad testning har visat på konkreta fördelar i tidigare forskning som utforskat omfattande områden. Clearingsystem inom finansmarknader är dock stora programvarusystem där dessa metoder sällan har utforskats. Denna arbete utforskar specifikt modellbaserad testning med användning av Modbat, ett specialiserat verktyg baserat på utökade ändliga automater, som ett kompletterande testningsmetodik för programvarurensningssystem på Nasdaq. Särskilt tillämpas metoden på specifika funktioner i systemet för att testa både dess förmåga att hitta buggar som inte upptäckts av traditionella metoder och att mäta dess effektivitet att skapa pålitlig täckning av kod på kort tid. I denna studie visar jag att den modellbaserade Modbat-metoden kan uppnå hög kodtäckning med relativt lite ansträngning. 77 % av produktionskoden täcktes med mycket få Modbat-testkörningar. Dessutom krävdes endast cirka 9-14 % av samma LOC, och en bråkdel av tiden för att uppnå dessa resultat.

SERVICE oriented architecture (SOA) is an architectural style to structure software systems, fostering loose coupling and dynamic integration among the applications. The use of SOA and Web services to develop complex and large business processes demands more formal and systematic testing. In addition, characteristics of this type of software limit the straightforward application of traditional testing techniques. Model-based testing (MBT) is a promising approach to deal with these problems. This dissertation investigates how two modeling techniques, namely Finite State Machine (FSM) and Event Sequence Graph (ESG), can be used to support MBT of service-oriented applications. Both techniques model different aspects and can be applied in a complementary way. Initially, we define an MBT process for service-oriented applications that employs FSMs. Based on previous experience, we propose a model-based approach to test composite services using ESGs. This approach is holistic, once test suites are generated to cover both desired situations (positive testing) and unexpected behaviors (negative testing). Three experimental studies evaluate the proposed approach: (i) a case study, (ii) a cost analysis, and (iii) a study in industry. Testing tools are also presented to support its practical use / A Arquitetura orientada a serviço (SOA) é um estilo arquitetural para estruturar sistemas de software de modo que exista um baixo grau de acoplamento entre as aplicações e essas possam ser facilmente integradas de forma dinâmica. A incorporação de SOA e serviços Web em sistemas que modelam processos de negócios grandes e complexos contribui para a necessidade de testes mais formais e sistemáticos. Além disso, características próprias dessa nova classe de software fazem com que técnicas de teste tradicionais não possam ser diretamente aplicadas. O teste baseado em modelo (TBM) apresenta-se como uma abordagem promissora que busca a resolução desses problemas. Esta tese investiga como duas técnicas de modelagem, Máquina de Estados Finitos (MEF) e Grafo de Sequência de Eventos (GSE), podem ser utilizadas para apoiar o TBM de aplicações orientadas a serviço. Essas técnicas modelam diferentes aspectos e podem ser aplicadas de forma complementar. Inicialmente, é definido um processo de TBM para aplicações orientadas a serviço que emprega MEFs. Com base na experiência adquirida, é proposta uma abordagem baseada em modelo para o teste de serviços compostos usando GSEs. Essa abordagem é holística uma vez que conjuntos de teste são gerados para cobrir tanto situações desejadas (teste positivo) quanto comportamentos inesperados (teste negativo). Três estudos experimentais avaliam a abordagem proposta: (i) um estudo de caso, (ii) uma análise de custo e (ii) um estudo na indústria. Ferramentas de teste também são apresentadas para apoiar o uso prático da abordagem proposta

A Arquitetura Orientada a Serviços (SOA) é um estilo arquitetural formado por um conjunto de restrições que visa promover a escalabilidade e a flexibilidade de um sistema, provendo suas funcionalidades como serviços. Nos últimos anos, um estilo alternativo foi proposto e amplamente adotado, que projeta as funcionalidades de um sistema como recursos. Este estilo arquitetural orientado a recursos é chamado de REST. O teste de serviços web em geral apresenta vários desafios devido a sua natureza distribuída, canal de comunicação pouco confiável, baixo acoplamento e a falta de uma interface de usuário. O teste de serviços RESTful (serviços que utilizam o REST) compartilham estes mesmos desafios e ainda necessitam que suas restrições sejam obedecidas. Estes desafios demandam testes mais sistemáticos e formais. Neste contexto, o teste baseado em modelos (TBM) se apresenta como um processo viável para abordar essas necessidades. O modelo que representa o sistema deve ser simples e ao mesmo tempo preciso para que sejam gerados casos de teste com qualidade. Com base nesse contexto, este projeto de mestrado propõe uma abordagem baseada em modelos para testar serviços RESTful. O modelo comportamental adotado foi a máquina de estados de protocolos UML, capaz de formalizar a interface do serviço enquanto esconde o seu funcionamento interno. Uma ferramenta foi desenvolvida para gerar automaticamente os casos de teste usando critérios de cobertura de estados e transições para percorrer o modelo / Service Oriented Architecture (SOA) is an architectural style consisting of a set of restrictions aimed at promoting the scalability and flexibility of a system, providing its functionalities as services. In recent years, an alternative style was proposed and widely adopted, which designs the system\'s functionalities as resources. This resource oriented architectural style is called REST. In general, the test of web services has several challenges due to its distributed nature, unreliable communication channel, low coupling and the lack of a user interface. Testing RESTful web services (services that use REST) share these same challenges and also need to obey the REST constraints. These challenges require a more systematic and formal testing approach. In this context, model based testing presents itself as a viable process for addressing those needs. The model that represents the system should be simple and precise enough to generate quality test cases. Based on this context, this work proposes a model based approach to test RESTful web services. The behavioral model used was the UML protocol state machine, which is capable to provide a formalization of the service interface, while hiding its internal behaviour. A tool was developed to automatically generate test cases using the state and transition coverage criteria to traverse the model

O Teste Baseado em Modelos (TBM) emergiu como uma estratégia promissora para minimizar problemas relacionados à falta de tempo e recursos em teste de software e visa verificar se a implementação sob teste está em conformidade com sua especificação. Casos de teste são gerados automaticamente a partir de modelos comportamentais produzidos durante o ciclo de desenvolvimento de software. Entre as técnicas de modelagem existentes, Sistemas de Transição com Entrada/Saída (do inglês, Input/Output Transition Systems - IOTSs), são modelos amplamente utilizados no TBM por serem mais expressivos do que Máquinas de Estado Finito (MEFs). Apesar dos métodos existentes para geração de testes a partir de IOTSs, o problema da seleção de casos de testes é um tópico difícil e importante. Os métodos existentes para IOTS são não-determinísticos, ao contrário da teoria existente para MEFs, que fornece garantia de cobertura completa com base em um modelo de defeitos. Esta tese investiga a aplicação de modelos de defeitos em métodos determinísticos de geração de testes a partir de IOTSs. Foi proposto um método para geração de conjuntos de teste com base no método W para MEFs. O método gera conjuntos de teste de forma determinística além de satisfazer condições de suficiência de cobertura da especificação e de todos os defeitos do domínio de defeitos definido. Estudos empíricos avaliaram a aplicabilidade e eficácia do método proposto: resultados experimentais para analisar o custo de geração de conjuntos de teste utilizando IOTSs gerados aleatoriamente e um estudo de caso com especificações da indústria mostram a efetividade dos conjuntos gerados em relação ao método tradicional de Tretmans. / Model-Based Testing (MBT) has emerged as a promising strategy for the minimization of problems related to time and resource limitations in software testing and aims at checking whether the implementation under test is in compliance with its specification. Test cases are automatically generated from behavioral models produced during the software development life cycle. Among the existing modeling techniques, Input/Output Transition Systems (IOTSs) have been widely used in MBT because they are more expressive than Finite State Machines (FSMs). Despite the existence of test generation methods for IOTSs, the problem of selection of test cases is an important and difficult topic. The current methods for IOTSs are non-deterministic, in contrast to the existing theory for FSMs that provides complete fault coverage guarantee based on a fault model. This manuscript addresses the application of fault models to deterministic test generation methods from IOTSs. A method for the test suite generation based on W method for FSMs is proposed for IOTSs. It generates test suites in a deterministic way and also satisfies sufficient conditions of specification coverage and all faults in a given fault domain. Empirical studies evaluated its applicability and effectiveness. Experimental results for the analyses of the cost of test suite generation by random IOTSs and a case study with specifications from the industry show the effectiveness of the test suites generated in relation to the traditional method of Tretmans.