The ability to test software is invaluable in all areas of computer science, but it is often neglected in computer science curricula. Test adequacy criteria (TAC), tools that measure the effectiveness of a test suite, have been used as aids to improve software testing teaching practices, but little is known about how students respond to them. Studies have examined the cognitive processes of students programming and professional developers writing tests, but none have investigated how student testers test with TAC. If we are to improve how they are used in the classroom, we must start by understanding the different ways that they affect students’ thought processes as they write tests. In this thesis, we take a grounded theory approach to reveal the underlying cognitive processes that students utilize as they test under no feedback, condition coverage, and mutation analysis. We recorded 12 students as they thought aloud while creating test suites under these feedback mechanisms, and then we analyzed these recordings to identify the thought processes they used. We present our findings in the form of the phenomena we identified, which can be further investigated to shed more light on how different TAC affect students as they write tests. i

Test suites for database applications depend on adequate test data and real-world test faults for success. An automated tool is available that quantifies test data coverage for database queries written in SQL. An automated tool is also available that mimics real-world faults by mutating SQL, however tests have revealed that these simulated faults do not completely represent real-world faults. This paper demonstrates how half of the mutation operators used by the SQL mutation tool in real-world test suites generated significantly lower detection scores than those from research test suites. Three revised mutation operators are introduced that improve detection scores and contribute toward re-defining a sufficient set of mutation operators for SQL. Finally, a procedure is presented that reduces the test burden by automatically comparing SQL mutants with their original queries. / Master of Science

Testing is an indispensable mechanism for assuring software quality. One of the key issues in testing is to obtain a test data set that is able to effectively test an implementation. An adequate test data set consists of test cases that can expose faults in a software implementation. Mutation-based testing can be employed to obtain adequate test data sets, and numerous mutation operators have been proposed to date to measure the adequacy of test data sets that reveal functional faults. However, implementations that pass functionality tests are still vulnerable to malicious attacks. Despite the rigorous use of various existing testing techniques, many vulnerabilities are discovered after the deployment of software implementations, such as buffer overflows (BOF), SQL injections, and format string bugs (FSB). Successful exploitations of these vulnerabilities may result in severe consequences such as denial of services, application state corruptions, and information leakage. Many approaches have been proposed to detect these vulnerabilities. Unfortunately, very few approaches address the issue of testing implementations against vulnerabilities. Moreover, these approaches do not provide an indication whether a test data set is adequate for vulnerability testing or not. We believe that bringing the idea of traditional functional test adequacy to vulnerability testing can help address the issue of test adequacy. In this thesis, we apply the idea of mutation-based adequate testing to perform vulnerability testing of buffer overflows, SQL injections, and format string bugs. We propose mutation operators to force the generation of adequate test data sets for these vulnerabilities. The operators mutate source code to inject the vulnerabilities in the library function calls and unsafe implementation language elements. The mutants generated by the operators are killed by test cases that expose these vulnerabilities. We propose distinguishing or killing criteria for mutants that consider varying symptoms of exploitations. Three prototype tools are developed to automatically generate mutants and perform mutation analysis with input test cases and the effectiveness of the proposed operators is evaluated on several open source programs containing known vulnerabilities. The results indicate that the proposed operators are effective for testing the vulnerabilities, and the mutation-based vulnerability testing process ensures the quality of the applications against these vulnerabilities. / Thesis (Master, Computing) -- Queen's University, 2008-08-18 13:53:04.036

Submitted by Lucienne Costa (lucienneferreira@ufcg.edu.br) on 2018-05-02T20:48:28Z No. of bitstreams: 1 ALAN KELON OLIVEIRA DE MORAES – TESE (PPGCC) 2017.pdf: 2206784 bytes, checksum: 9e05994ae273d28b9871d8d16769dac8 (MD5) / Made available in DSpace on 2018-05-02T20:48:28Z (GMT). No. of bitstreams: 1 ALAN KELON OLIVEIRA DE MORAES – TESE (PPGCC) 2017.pdf: 2206784 bytes, checksum: 9e05994ae273d28b9871d8d16769dac8 (MD5) Previous issue date: 2017-08-31 / O teste baseado em modelos é uma abordagem de teste de software que usa modelos abstratos de uma aplicação para gerar, executar e avaliar os testes. A geração de casos de testes exerce um papel importante no teste baseado em modelos. Como essa geração consiste na busca sistemática por casos de testes que possam ser extraídos dos modelos, o teste baseado em modelos geralmente produz suítes de testes que são caras demais para serem executadas completamente. Técnicas de redução de suítes de testes têm sido propostas para abordar este problema. O objetivo dessas técnicas é obter suítes de testes reduzidas que são mais baratas de serem executadas e tão efetivas na detecção de faltas quanto as suítes completas, dado que as suítes reduzidas mantém o mesmo nível de cobertura, definido por um critério de adequação de testes, da suíte completa. Esses critérios definem que partes do sistema serão testados, com que frequência e sob quais circunstâncias. Entretanto, pouca atenção tem sido dada ao impacto que a escolha do critério tem na redução de suítes de testes. Por outro lado, sistemas de tempo-real são sistemas reativos cujos comportamentos são restringidos pelo tempo. Consequentemente, faltas relacionadas ao tempo são específicas desses sistemas. Para lidar com isso, modelos para sistemas de tempo real devem trabalhar com tempo e, consequentemente, há critérios de adequação de testes específicos para eles. Contudo, a pesquisa sobre redução de suítes de testes não tem focado em sistemas de tempo-real, portanto o impacto de critérios de adequação de testes na redução de suítes é desconhecido. Nesta pesquisa de doutorado objetivamos investigar a influência de critérios de adequação de testes nos resultados da redução de suítes de testes no contexto de teste baseado em modelos de sistemas de tempo-real. Em particular, nós estamos interessados no modelo Timed Input-Output Symbolic Transition Systems (TIOSTS), porque ele é um modelo de sistema de transições no qual dados e tempo são definidos simbolicamente, já que sistemas de transição são a base para o teste de conformidade de sistemas de tempo real. Para alcançar o objetivo da pesquisa, primeiramente, nós definimos 19 critérios de adequação de testes para o modelo TIOSTS. Os critérios definidos incluem critérios baseados em transições, fluxo de dados e tempo. Depois nós formalizamos uma hierarquia com esses critérios, onde eles estão parcialmente ordenados pela relação de inclusão estrita. Segundamente, nós avaliamos empiricamente o custo-benefício de doze dos critérios definidos e cinco técnicas de redução de suítes de testes. Nós avaliamos o tamanho, o tempo de execução e a detecção de faltas das suítes de testes reduzidas de cada uma das 60 combinações de critério e técnica. No experimento, nós usamos modelos de especificação, em TIOSTS, de uma máquina de recarga de cartão do metrô, de um sistema de alarme anti-roubo e de um limitador automático de velocidade de carros. Além disso, usamos simulações das implementações, que geram rastros corretos para os modelos. Por fim, o teste de mutação foi usado para gerar mutantes dos modelos de especificação, que, por sua vez, foram traduzidos para simulações com a finalidade de simular modelos de implementações defeituosas. As evidências empíricas sugerem que os critérios de adequação de testes mais próximos do topo da hierarquia produziram suítes reduzidas com melhor custo-benefício com relação à detecção de faltas e tempo de execução. Com relação às técnicas de redução, a técnica aleatória obteve melhor custo-benefício dentre as técnicas avaliadas. Os resultados apontam que os critérios explicam mais a variação nos resultados do que as técnicas. / Model-based testing is a testing approach that relies on the existence of abstract models of an application to generate, execute and evaluate tests. Test case generation plays an important role in model-based testing. Since it consists of a systematic search for test cases that can be extracted from models, model-based testing usually generates large test suites which are too expensive to execute in full. Test suite reduction techniques have been proposed to address this problem. The goal of the techniques is to obtain reduced test suites that are both cheaper to execute and as effective at detecting faults as the original suite, given that the reduced test suites maintain the same coverage level of the complete test suite required by a test adequacy criterion. These criteria define which parts of the system are going to be tested, how often and under what circumstances. Nevertheless, little attention has been paid to the impact of the criterion choice in test suite reduction research. On the other hand, real-time systems are reactive systems whose behavior is constrained by time. Consequently, time-related faults are specific to these systems. In order to cope with this issue, models for real-time systems must deal with time and, consequently, there are specific test adequacy criteria for them. However, test suite reduction research has not focused on real-time systems, therefore the impact of test adequacy criteria for models of real-time systems on test suite reduction is unknown. In this doctoral research, we aim at investigating the influence of test adequacy criteria on the outcomes of test suite reduction techniques in the context of model-based testing of real-time systems. In particular, we are interested in the Timed Input-Output Symbolic Transition Systems (TIOSTS) model because it is an expressive transition system in which data and time are symbolically defined, and transition systems are the basis for conformance testing of real-time systems. In order to achieve the research objective, first, we defined 19 test adequacy criteria for TIOSTS models. The defined criteria include transition-based criteria, data-flow-oriented criteria and real-time systems criteria. Next, we formalized a hierarchy with these criteria which is partially ordered by the strict inclusion relation. Second, we evaluated the cost-effectiveness of twelve criteria and five test suite reduction techniques in empirical studies of test suite reduction. We evaluated the size, execution time and fault detection of reduced test suites obtained from each combination of criterion and technique. In the experiment, we used TIOSTS specification models of a refilling machine for charging the subway card, a burglar alarm system, and an automated car speed limiter; simulations of the implementations, which generate correct traces for the models; and mutation testing to generate mutants of the specification models, which were also translated to simulations in order to simulate faulty model implementations. Empirical evidence suggests that test adequacy criteria closer to the top of the family obtained reduced test suites with better costeffectiveness regarding fault detection and execution time. With respect to the test suite reduction techniques, the Random technique obtained better cost-effectiveness among the evaluated criteria. Results also suggests that the criteria explain more the variation in fault detection and execution time of reduced test suites than the techniques.