PROCESS algebras are a family of techniques used in formal specification and analysis of computer systems, specially when independent processes that perform and synchronize in parallel are concerned. The so-called concurrent systems. Circus is a process algebra that aggregates the expressiveness power for concurrent behaviors from CSP, along with the predicative data modelling aspects of Z. Recent publications have established a formal exhaustive symbolic testing theory for specifications modelled in Circus. Aiming to improve the feasibility of applying such tests in practical scenarios, it is convenient to look for criteria that reduces the number of test cases which, by the exhaustive nature of the approach, is often infinite. In the light of this, the work we present proposes the application of mutation testing techniques in Circus specifications, targeting the coverage of faults seeded by well-known mutation operators, along with operators designed with the particularities of the language in mind. Some contributions were produced in the pursuit of these goals, such as establishing a formal theory for mutation testing in Circus specifications and the implementation of a symbolic traces generator for the language. / ÁLGEBRAS de processos são uma família de técnicas de especificação e análise formal utilizadas em sistemas computacionais, especialmente em contextos de processos independentes, que atuam paralelamente e efetuam comunicação entre si. São os chamados sistemas concorrentes. Circus é uma álgebra de processos que agrega a capacidade de expressão de comportamentos concorrentes do CSP com a modelagem predicativa de dados da notação Z. Trabalhos recentes vêm estabelecer uma teoria para o teste simbólico exaustivo baseado em especificações modeladas em Circus. Com o objetivo de viabilizar a aplicação prática desses testes, é conveniente estudar critérios que reduzam o conjunto de casos de teste que, pela sua natureza exaustiva, torna-se frequentemente infinito. Neste sentido, o presente trabalho propõe a aplicação de técnicas de teste de mutação à partir de especificações Circus, visando a cobertura de falhas inseridas por meio de operadores de mutação já conhecidos, juntamente com operadores propostos especificamente para a linguagem. Algumas contribuições foram produzidas na busca destes objetivos, como o estabelecimento de uma teoria formal para a aplicação de teste de mutação em especificações Circus e a implementação de um gerador de rastros simbólicos para a mesma linguagem.

A problem when testing timeliness of event-triggered real-time systems is that response times depend on the execution order of concurrent tasks. Conventional testing methods ignore task interleaving and timing and thus do not help determine which execution orders need to be exercised to gain confidence in temporal correctness. This thesis presents and evaluates a framework for testing of timeliness that is based on mutation testing theory. The framework includes two complementary approaches for mutation-based test case generation, testing criteria for timeliness, and tools for automating the test case generation process. A scheme for automated test case execution is also defined. The testing framework assumes that a structured notation is used to model the real-time applications and their execution environment. This real-time system model is subsequently mutated by operators that mimic potential errors that may lead to timeliness failures. Each mutated model is automatically analyzed to generate test cases that target execution orders that are likely to lead to timeliness failures. The validation of the theory and methods in the proposed testing framework is done iteratively through case-studies, experiments and proof-of-concept implementations. This research indicates that an adapted form of mutation-based testing can be used for effective and automated testing of timeliness and, thus, for increasing the confidence level in real-time systems that are designed according to the event-triggered paradigm.

Mutation testing has traditionally been used to evaluate the effectiveness of test suites and provide con dence in the testing process. Mutation testing involves the creation of many versions of a program each with a single syntactic fault. A test suite is evaluated against these program versions (i.e., mutants) in order to determine the percentage of mutants a test suite is able to identify (i.e., mutation score). A major drawback of mutation testing is that even a small program may yield thousands of mutants and can potentially make the process cost prohibitive. To improve the performance and reduce the cost of mutation testing, we proposed a machine learning approach to predict mutation score based on a combination of source code and test suite metrics. We conducted an empirical evaluation of our approach to evaluated its effectiveness using eight open source software systems. / UOIT

Šiame magistro darbe pateikiama automatinių testų generavimas XML WEB servisams, naudojantis WSDL failais. Pirma dalis aprašo algoritmų tyrimą ir būdus generuoti automatinius testus XML WEB servisams. Buvo sukurtas papildinys Visual Studio programinės įrangos kūrimo įrankiui. Papildinys, pasinaudojant WSDL specifikacija, analizuoja ir atvaizduoja WEB serviso struktūrą medyje. Vartotojai, naudojantis išanalizuotais duomenimis, gali lengvai įvesti testavimo duomenis ir generuoti automatinius vienetų testus. Taip pat sistema leidžia vartotojui įvesti atsako laiką ir generuoti automatinius našumo testus. Aprašyti papildinio reikalavimai, funkcinė ir nefunkcinė specifikacija bei architektūra. Tyrimo skyriuje yra aprašomas sukurto įrankio tyrimas. Šioje srityje buvo tiriama darbo su įrankiu efektingumas, atliekant mutacinį testavimą. / In this master thesis the automated tests generation for XML WEB services by using WSDL files is presented. First section describes a research of algorithms and ways to generate automated test for XML WEB services. For automated tests generation plug-in for Visual Studio development tool is created. Plug-in takes web service specification analyses and shows WEB service structure in tree. Users by using analyzed data user can easily enter testing data by using developed tool and generate automated unit tests. Also system let user to enter response time and generate performance tests. Created plug-in requirements, functional and non-functional specification, architecture. In investigation section is described developed plug-in investigation. In this section were investigate the working efficiency of tool by doing mutation testing for generated automated tests.

Mobility occurs naturally in many distributed system applications such as telecommunications and electronic commerce. Mobility may reduce bandwidth consumption and coupling and increase flexibility. However, it seems that relatively little work has been done to support quality assurance techniques such as testing and verification of mobile systems. This thesis describes an approach for checking the conformance of a mobile, distributed application with respect to an executable model at runtime. The approach is based on kiltera -- a novel, high-level language supporting the description and execution of models of concurrent, mobile, distributed, and timed computation. The approach allows distributed, rather than centralized, monitoring. However, it makes very few assumptions about the platform that the mobile agent system is implemented in. We have implemented our approach and validated it using four case studies. Two of them are examples of mobile agent systems, the two others are implementations of distributed algorithms. Our approach was able to detect seeded faults in the implementations. To check the effectiveness and the efficiency of our approach more comprehensively a mutation-based evaluation framework has been implemented. In this framework a set of a new mutation operators for mobile agent systems has been identified in order to automatically generate and run a number of mutants programs and then evaluate the ability of our approach to detect these mutants. We found that our approach is very effective and efficient in killing the non-equivalent mutants. / Thesis (Ph.D, Computing) -- Queen's University, 2010-04-27 12:35:47.996

A Programação Concorrente tornou-se uma forma popular de desenvolvimento de software. Este paradigma de desenvolvimento e essencial para construir aplicações com o intuito de reduzir o tempo computacional em muitos domínios como, por exemplo, previsão tempo, processamento de imagem, entre outros. Estes programas têm novas características como a comunicação, a sincronização e o não determinismo, que precisam ser considerados durante a atividade de teste. O teste de software e uma atividade que busca garantir a qualidade por meio da identificação de falhas no produto. O Teste de Mutação e um critério de teste que se baseia nos enganos que podem ser cometidos pelos desenvolvedores de software. Porém, o teste de mutação não pode ser aplicado em programas concorrentes da mesma maneira como e aplicado em programas sequenciais por causa das particularidades presentes nos programas concorrentes. Um problema de aplicar o teste de mutação nesse contexto e o comportamento não determinístico das aplicações. Este trabalho investiga a definição do teste de mutação para programas concorrentes implementados em MPI (Message Passing Interface), os quais realizam comunicação e sincronização por meio de troca de mensagens. Para isso, defeitos típicos nesse domínio foram considerados, buscando modelar operadores de mutação para tratar os aspectos de comunicação e sincronização dessas aplicações. Também foi proposto um procedimento para dar suporte a análise comportamental dos mutantes. As idéias foram implementadas em uma ferramenta de teste chamada ValiMPI Mut / Concurrent programming became a popular paradigm for software development. This paradigm is essential to build applications which aim to reduce the computational time in many areas, such as, weather forecast, image processing, among others. These programs present new features such as communication, synchronization, and nondeterminism, which must be considered during the testing activity. Software testing is an activity that looks to ensure quality by identifying faults in the product. Mutation Testing is a criterion based on the most common mistakes that might be made by software developers. However, the mutation testing cannot be applied in concurrent programs the same way as applied in sequential ones due to the peculiarities present in concurrent programs. One of the problems in applying mutation testing in this context is the non-deterministic behavior. This work investigates the definition of mutation testing for concurrent programs implemented in MPI (Message Passing Interface), which perform communication and synchronization using message passing. For this, typical faults in this area were considered in order to model mutation operators addressing the aspects of communication and synchronization of these applications. Also, we are proposing a new procedure to support the behavioral analysis of the mutants. The ideas were implemented in a testing tool called ValiMPI Mut

No desenvolvimento de software, a qualidade do produto está diretamente relacionada à qualidade do processo de desenvolvimento. Diante disso, atividades de Verificação, Validação & Teste (VV&T) realizadas por meio de métodos, técnicas e ferramentas são de extrema necessidade para o aumento da produtividade, qualidade e diminuição de custos no desenvolvimento de software. Do mesmo modo, técnicas e critérios contribuem para a produtividade das atividades de teste. Um ponto crucial para o teste de software é sua automatização, tornando as atividades mais confiáveis e diminuindo significativamente os custos de desenvolvimento. Na automatização dos testes, os oráculos são essenciais, representando um mecanismo (programa, processo ou dados) que indica se a saída obtida para um caso de teste está correta. Este trabalho de mestrado utiliza a ideia de mutação para criar implementações alternativas de oráculos de teste e, assim, avaliar a sua qualidade. O teste de mutação se refere à criação de versões do sistema em desenvolvimento com pequenas alterações sintáticas de código. A mutação possui alta eficácia na detecção de defeitos e é bastante flexível na sua aplicação, podendo ser utilizada em diversos tipos de artefatos. Adicionalmente, este trabalho propõe operadores de mutação específicos para oráculos, implementa uma ferramenta de apoio à utilização desses operadores para oráculos e também descreve um estudo empírico dos operadores, destacando benefícios e desafios associados ao seu uso. / In software development, product quality is directly related to the quality of the development process. Therefore, activities of Verification, Validation & Testing (VV&T) performed by methods, techniques and tools are urgently required to increase productivity, quality and cost reduction in software development. Similarly, testing technique and criteria contribute to the productivity of test activities. A crucial point for the software testing automation is making the most reliable activities and significantly reducing development costs. Regarding software testing automation, test oracles are essential, representing an mechanism (program, process or data) to indicate whether the actual output for a given test case is correct. This masters thesis aims to explore concepts of mutation testing to create alternative implementations of the oracle procedure and thus assess their quality. Mutation testing refers to the creation of system development versions with minor syntactic code changes. It has high efficiency on defects detecting and it is very flexible in its application and it is being used in various types of artifacts. This work also proposes specific mutation operators for oracles, implements an useful support tool for using these oracle mutation operators and conducts an empirical study of operators, highlighting benefits and challenges associated with their use.

Um dos pontos fundamentais na atividade de teste de software é o projeto de casos de teste. Diversos critérios de adequação têm sido propostos com o objetivo de fornecer meios que permitam que a avaliação e elaboração de casos de teste sejam feitas de maneira sistemática e fundamentadas teoricamente. Infelizmente, a maioria dos critérios de adequação de casos de teste definidos tem seu uso restrito ao teste de unidade. Para fases posteriores da atividade de teste, em particular para o teste de integração, nota-se a ausência de critérios de adequação, principalmente porque os critérios propostos definem requisitos de teste que se restringem aos limites de uma única unidade, não exercitando de maneira efetiva as interações entre as unidades, que devem ser alvo principal no teste de integração. Com exceção de alguns poucos trabalhos que procuram estender critérios estruturais para o nível interprocedimental, tem-se utilizado nessa fase de teste, quase que exclusivamente, critérios funcionais. Dada essa ausência de critérios e salientando ainda o caráter complementar entre as diferentes técnicas de teste, esta tese apresenta um critério de teste interprocedimental baseado em defeitos chamado de Mutação de Interface. Esse critério busca exercitar as interações entre as unidades através da seleção de casos de teste que distingam mutantes criados pela introdução de defeitos típicos e que, de acordo com um modo definido, caracterizamos erros de integração. Definiu-se um conjunto de operadores de Mutação de Interface que concentram sua aplicação em pontos do programa relacionados com as interações entre as unidades, como, por exemplo, chamadas de subprogramas e seus parâmetros. Dados o alto custo de aplicação, inerente de critérios baseados em mutação, e pelas próprias características do conjunto de operadores de Mutação de Interface, torna-se necessário definirem-se abordagens para reduzir esse custo. Assim, foram estabelecidas maneiras de se parametrizar a aplicação dos operadores de mutação, definindo-se critérios de Mutação de Interface alternativos, estendendo-se abordagem sutilizadas no teste de mutação convencional como mutação restrita. A aplicação de um critério de teste está fortemente condicionada à sua automatização. A definição de um critério de teste sem que pelo menos se apontem soluções para sua automatização tem pouca utilidade prática. Por isso, especificou-se e implementou-se a ferramenta PROTEUM/IM para apoiar a aplicação do critério Mutação de Interface. Essa ferramenta torna-se essencial neste trabalho à medida que permite que estudos empíricos possam ser realizados, avaliando o critério proposto. Dois estudos de caso são apresentados. Esses estudos aplicam o critério Mutação de interface em programas reais e buscam avaliar seu custo e sua eficácia em revelar erros. Estes estudos aplicam ainda critérios alternativos, mostrando que a Mutação de interface é bastante efetiva em revelar erros o de ter custo de aplicação bastante reduzido, quando aplicada de maneira incremental, utilizando-se as parametrizações que os operadores de mutação oferecem. / The project of test cases is one of the most important topics in the software testing activity. Several criteria have been proposed aiming at allowing the evaluation and selection of test cases in a systematic and theoretically well founded way. Unfortunately, the use of most of these criteria is restricted to the unit testing phase. For other testing phases, in particular for integration testing. there is a lack of such criteria, mainly because the existing criteria define test requirements only in the scope of a single unit. They arc not able to effectively exercise the interactions between units, what should be the focus of integration testing. Excepting some few works that extend structural criteria to the interprocedural level, only functional testing has been used at integration testing phase. Given this lack of criteria and the complementary characteristics of different testing techniques, this thesis presents an interprocedural fault based criterion named Interface Mutation. This criterion exercises the interactions between units through the selection of test cases that distinguish mutants created by introducing typical faults that characterize integration errors. A set of Interface Mutation operators was defined. The focus of these operators are the points of the program related to the unit interactions, for instance, subprogram calls and their parameters. Given the high cost associated to mutation testing in general and particularly to the Interface Mutation operators, it is necessary to define some approaches to reduce its application cost. Thus, some parameterizations were defined to the mutation operators, allowing to establish alternative Interface Mutation criteria, extending approaches already used in conventional mutation testing, as random mutation and constrained mutation. The application of any testing criterion strongly depends on its automatization. The definition of a criterion, without pointing out ways to its implementation has little practical utility. So, a tool named PROTEUM/IM was specified and implemented to support the application of Interface Mutation. This tool is an essential point in the present work because it allows the conduction of empirical studies aiming at evaluating the proposed criterion. Two case studies arc presented. In these studies the criterion Interface Mutation is applied to real programs and the cost of its application as well as its errors revealing effectiveness are evaluated. Alternative criteria are also used. Showing that Interface Mutation is very effective to reveal errors and can be applied with a reduced cost if used in an incremental way, taking advantage of the parameterization characteristics provided by the Interface Mutation operators set.

Telecom base stations are a critical part of society's information infrastructure. To ensure high quality base station software, automated testing is an important part of development. Ericsson measures the quality of automated tests with statement coverage, counting the number of statements executed by a test suite. Alone, however, statement coverage does not guarantee test quality. Mutation testing is a technique to improve test quality by injecting faults and verifying that test suites detect them. This thesis investigates whether mutation testing is a viable way to increase the reliability of test suites for base station software at Ericsson. Using the open-source mutation testing tool MiLu, we describe a practical method of using mutation testing that is viable for daily development. We also describe how mutation testing reveals a numbers of potential errors in the production code that current test suites miss even though they have very good statement coverage.

For avionic systems, the safety requirements are stricter than for non-safety critical systems due to the severe consequences a failure could cause. Depending on the consequences of a failure, the software needs to fulfill different testing criterias. More critical software needs more extensive testing. The question is whether the extra testing activities performed for software of higher criticality level results in discovery of more faults. Mutation testing has been used in this thesis as a method to evaluate the quality of test suites of avionic applications from different safety critical levels. The results showed that the extra activities performed at the higher levels do not necessarily result in finding more faults.