Context: The importance of testing non-functional properties of the system is growing steadily. Complexity factor of the software is growing proportionally with the growing demands and hence attributes like performance, energy consumption and reliability are proving to be very crucial. Optimizing the software with respect to these properties simultaneously with the functional properties has been found to be a challenge. Search-based Software testing automates this process by using different meta-heuristic techniques. It assures the generation of large number of test cases at a minimal cost. Carrying out testing in this context requires lot of expertise and the aid of a highly flexible approach. There is a strong need of a guide that helps the practitioners(testers) and researchers optimize the non-functional properties using Search-based software testing. Objectives: The objective of the work presented in this thesis is to, first, investigate the non-functional properties, challenges encountered and approaches/suggestions by the practitioners on the application of Search-based software testing in academia and industry. Second objective is to map all the information into a conceptual/ theoretical framework that could be used by Search-based software testing practitioners for optimizing the non-functional system properties. Methods: A qualitative approach has been employed for this thesis work. A literature review with snowball sampling as the search approach was conducted to collect the information regarding the different kinds of systems being tested, the non-functional system properties being optimized, challenges encountered and the tools used for this purpose. Semi-structured interviews are conducted as a part of the validation process and generalizing the results obtained. A total of 9 interviews were conducted. Thematic analysis technique has been used to analyze the collected data. Results: As a result of conducting this research, different dimensions forming the framework have been investigated. The overall result of this study is the formulation fo a framework and that has been validated by conducting interviews. The framework consists of 16 challenges related to the field of Non-functional Search-based software testing. Conclusions: It is found out that Search-based testing for non-functional properties has not been extensively applied in the industry. It has been suggested, used and applied in academia for the most part. Several factors influence the selection of non-functional properties for optimization. Most of the challenges being faced in this subject are inclined towards three areas in Search-based testing. Performance, execution time and energy consumption are three most popularly tested attributes. Further research could be done wherein the framework generated could be put to use by different practitioners and researchers to find out interesting things.

No description available.

Manually creating test cases is time consuming and error prone. Search-based software testing (SBST) can help automate this process and thus to reduce time and effort and increase quality by automatically generating relevant test cases. Previous research have mainly focused on static programming languages with simple test data inputs such as numbers. In this work we present an approach for search-based software testing for dynamic programming languages that can generate test scenarios and both simple and more complex test data. This approach is implemented as a tool in and for the dynamic programming language Ruby. It uses an evolutionary algorithm to search for tests that gives structural code coverage. We have evaluated the system in an experiment on a number of code examples that differ in complexity and the type of input data they require. We compare our system with the results obtained by a random test case generator. The experiment shows, that the presented approach can compete with random testing and, for many situations, quicker finds tests and data that gives a higher structural code coverage.

Concurrent programming has become an essential paradigm for reductions in the computational time in many application domains. However, the validation and testing activity is more complex than the testing for sequential programs due to the non-determinism, synchronization and inter-process communication. Mutation testing is based on mistakes produced by software developers and presents a high effectiveness to reveal faults. However, high computational costs limit its applicability even for sequential code, becoming higher for concurrent programs in which each test has to be executed with different (ideally all) thread schedules. To date, only selective mutation have been applied to reduce the number of mutants in concurrent programs, however, the problem of state explosion of thread schedules still remains. This Ph.D. thesis presents the SBBMuT approach that applies deterministic execution and genetic algorithm for the generation and execution of a set of synchronization sequences during the mutation testing of Java multithreaded programs. An experimental study was conducted, and the results showed that the set of synchronization sequences generated by SBBMuT achieved a higher mutation score in comparison with the use of the Java PathFinder model checking tool. / A programação concorrente tornou-se um paradigma essencial para a redução no tempo computacional em muitos domínios de aplicação. No entanto, as atividades de verificação, validação e teste são mais complexas do que o teste para programas sequenciais devido ao não determinismo, sincronização e comunicação entre processos ou threads. O teste de mutação é baseado em enganos cometidos por desenvolvedores de software e apresenta uma alta eficácia para revelar defeitos. No entanto, o alto custo computacional limita a sua aplicação mesmo para programas sequenciais, e tornando-se maior para programas concorrentes no qual cada teste deve ser executado com diferentes (idealmente todas) sequências de sincronizações. Na literatura, apenas mutação seletiva foi aplicada para reduzir o número de mutantes em programas concorrentes, no entanto, o problema de explosão no número de sequências de sincronização ainda permanece. Esta tese de doutorado apresenta a abordagem SBBMuT que aplica execução determinística e algoritmo genético para a geração e execução de um conjunto de sequências de sincronização durante o teste de mutação para programas Java multithread. Um estudo experimental foi conduzido e os resultados mostram que o conjunto de sequências de sincronização gerada pela SBBMuT conseguiu alcançar um escore de mutação maior em comparação com a utilização da ferramenta de validação de modelos Java PathFinder.

Submitted by Marlene Santos (marlene.bc.ufg@gmail.com) on 2016-06-09T17:02:10Z No. of bitstreams: 2 Tese - Eduardo Noronha de Andrade Freitas - 2016.pdf: 1936673 bytes, checksum: 4336d187b0e552ae806ef83b9f695db0 (MD5) license_rdf: 19874 bytes, checksum: 38cb62ef53e6f513db2fb7e337df6485 (MD5) / Approved for entry into archive by Luciana Ferreira (lucgeral@gmail.com) on 2016-06-10T11:14:00Z (GMT) No. of bitstreams: 2 Tese - Eduardo Noronha de Andrade Freitas - 2016.pdf: 1936673 bytes, checksum: 4336d187b0e552ae806ef83b9f695db0 (MD5) license_rdf: 19874 bytes, checksum: 38cb62ef53e6f513db2fb7e337df6485 (MD5) / Made available in DSpace on 2016-06-10T11:14:00Z (GMT). No. of bitstreams: 2 Tese - Eduardo Noronha de Andrade Freitas - 2016.pdf: 1936673 bytes, checksum: 4336d187b0e552ae806ef83b9f695db0 (MD5) license_rdf: 19874 bytes, checksum: 38cb62ef53e6f513db2fb7e337df6485 (MD5) Previous issue date: 2016-02-15 / Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES / Fundação de Amparo à Pesquisa do Estado de Goiás - FAPEG / The creation of a suite of unit testing is preceded by the selection of which components (code units) should be tested. This selection is a significant challenge, usually made based on the team member’s experience or guided by defect prediction or fault localization models. We modeled the selection of components for unit testing with limited resources as a multi-objective problem, addressing two different objectives: maximizing benefits and minimizing cost. To measure the benefit of a component, we made use of important metrics from static analysis (cost of future maintenance), dynamic analysis (risk of fault, and frequency of calls), and business value. We tackled gaps and challenges in the literature to formulate an effective method, the Selector of Software Components for Unit testing (SCOUT). SCOUT was structured in two stages: an automated extraction of all necessary data and a multi-objective optimization process. The Android platform was chosen to perform our experiments, and nine leading open-source applications were used as our subjects. SCOUT was compared with two of the most frequently used strategies in terms of efficacy.We also compared the effectiveness and efficiency of seven algorithms in solving a multi-objective component selection problem: random technique; constructivist heuristic; Gurobi, a commercial tool; genetic algorithm; SPEA_II; NSGA_II; and NSGA_III. The results indicate the benefits of using multi-objective evolutionary approaches such as NSGA_II and demonstrate that SCOUT has a significant potential to reduce market vulnerability. To the best of our knowledge, SCOUT is the first method to assist software testing managers in selecting components at the method level for the development of unit testing in an automated way based on a multi-objective approach, exploring static and dynamic metrics and business value. / (Sem resumo)

Concurrent programming has become an essential paradigm for reductions in the computational time in many application domains. However, the validation and testing activity is more complex than the testing for sequential programs due to the non-determinism, synchronization and inter-process communication. Mutation testing is based on mistakes produced by software developers and presents a high effectiveness to reveal faults. However, high computational costs limit its applicability even for sequential code, becoming higher for concurrent programs in which each test has to be executed with different (ideally all) thread schedules. To date, only selective mutation have been applied to reduce the number of mutants in concurrent programs, however, the problem of state explosion of thread schedules still remains. This Ph.D. thesis presents the SBBMuT approach that applies deterministic execution and genetic algorithm for the generation and execution of a set of synchronization sequences during the mutation testing of Java multithreaded programs. An experimental study was conducted, and the results showed that the set of synchronization sequences generated by SBBMuT achieved a higher mutation score in comparison with the use of the Java PathFinder model checking tool. / A programação concorrente tornou-se um paradigma essencial para a redução no tempo computacional em muitos domínios de aplicação. No entanto, as atividades de verificação, validação e teste são mais complexas do que o teste para programas sequenciais devido ao não determinismo, sincronização e comunicação entre processos ou threads. O teste de mutação é baseado em enganos cometidos por desenvolvedores de software e apresenta uma alta eficácia para revelar defeitos. No entanto, o alto custo computacional limita a sua aplicação mesmo para programas sequenciais, e tornando-se maior para programas concorrentes no qual cada teste deve ser executado com diferentes (idealmente todas) sequências de sincronizações. Na literatura, apenas mutação seletiva foi aplicada para reduzir o número de mutantes em programas concorrentes, no entanto, o problema de explosão no número de sequências de sincronização ainda permanece. Esta tese de doutorado apresenta a abordagem SBBMuT que aplica execução determinística e algoritmo genético para a geração e execução de um conjunto de sequências de sincronização durante o teste de mutação para programas Java multithread. Um estudo experimental foi conduzido e os resultados mostram que o conjunto de sequências de sincronização gerada pela SBBMuT conseguiu alcançar um escore de mutação maior em comparação com a utilização da ferramenta de validação de modelos Java PathFinder.

O teste de mutação é um critério de teste poderoso para detectar falhas e medir a eficácia de um conjunto de dados de teste. No entanto, é uma técnica de teste computacionalmente cara. O alto custo provém principalmente do esforço para gerar dados de teste adequados para matar os mutantes e pela existência de mutantes equivalentes. Nesse contexto, o objetivo desta tese é apresentar uma abordagem chamada de Reach, Infect and Propagation to Mutation Testing (RIPMuT) que visa gerar dados de teste e sugerir mutantes equivalentes. A abordagem é composta por dois módulos: (i) uma geração automatizada de dados de teste usando subida da encosta e um esquema de fitness de acordo com as condições de alcançabilidade, infeção e propagação (RIP); e (ii) um método para sugerir mutantes equivalentes com base na análise das condições RIP durante o processo de geração de dados de teste. Os experimentos foram conduzidos para avaliar a eficácia da abordagem RIP-MuT e um estudo comparativo com o algoritmo genético e testes aleatórios foi realizado. A abordagem RIP-MuT obteve um escore médio de mutação de 18,25 % maior que o AG e 35,93 % maior que o teste aleatório. O método proposto para detecção de mutantes equivalentes se mostrou viável para redução de custos relacionado a essa atividade, uma vez que obteve uma precisão de 75,05% na sugestão dos mutantes equivalentes. Portanto, os resultados indicam que a abordagem gera dados de teste adequados capazes de matar a maioria dos mutantes em programas C e, também auxilia a identificar mutantes equivalentes corretamente. / Mutation Testing is a powerful test criterion to detect faults and measure the effectiveness of a test data set. However, it is a computationally expensive testing technique. The high cost comes mainly from the effort to generate adequate test data to kill the mutants and by the existence of equivalent mutants. In this thesis, an approach called Reach, Infect and Propagation to Mutation Testing (RIP-MuT) is presented to generate test data and to suggest equivalent mutants. The approach is composed of two modules: (i) an automated test data generation using hill climbing and a fitness scheme according to Reach, Infect, and Propagate (RIP) conditions; and (ii) a method to suggest equivalent mutants based on the analyses of RIP conditions during the process of test data generation. The experiments were conducted to evaluate the effectiveness of the RIP-MuT approach and a comparative study with a genetic algorithm and random testing. The RIP-MuT approach achieved a mean mutation score of 18.25% higher than the GA and 35.93% higher than random testing. The proposed method for detection of equivalent mutants demonstrate to be feasible for cost reduction in this activity since it obtained a precision of 75.05% on suggesting equivalent mutants. Therefore, the results indicate that the approach produces effective test data able to strongly kill the majority of mutants on C programs, and also it can assist in suggesting equivalent mutants correctly.

Submitted by Marlene Santos (marlene.bc.ufg@gmail.com) on 2014-10-15T17:49:53Z No. of bitstreams: 2 Dissertacao - Ana Claudia Bastos Loureiro Monção - 2013.pdf: 4213405 bytes, checksum: 3bbe190ae0f4a45a2f8b4e71026f5d2e (MD5) license_rdf: 23148 bytes, checksum: 9da0b6dfac957114c6a7714714b86306 (MD5) / Approved for entry into archive by Jaqueline Silva (jtas29@gmail.com) on 2014-10-16T17:59:00Z (GMT) No. of bitstreams: 2 Dissertacao - Ana Claudia Bastos Loureiro Monção - 2013.pdf: 4213405 bytes, checksum: 3bbe190ae0f4a45a2f8b4e71026f5d2e (MD5) license_rdf: 23148 bytes, checksum: 9da0b6dfac957114c6a7714714b86306 (MD5) / Made available in DSpace on 2014-10-16T17:59:00Z (GMT). No. of bitstreams: 2 Dissertacao - Ana Claudia Bastos Loureiro Monção - 2013.pdf: 4213405 bytes, checksum: 3bbe190ae0f4a45a2f8b4e71026f5d2e (MD5) license_rdf: 23148 bytes, checksum: 9da0b6dfac957114c6a7714714b86306 (MD5) Previous issue date: 2013-08-02 / Software Testing is an important area of Software Engineering to ensuring the software quality. It consists of activities that involve long time and high costs, but need to be made throughout the process of building software. As in other areas of software engineering, there are problems in the activities of Software Testing whose solution is not trivial. For these problems, several techniques of optimization and search have been explored trying to find an optimal solution or near optimal, giving rise to lines of research textit Search-Based Software Engineering (SBSE) and textit Search-Based Software Testing (SBST). This work is part of this context and aims to solve the problem of selecting test data for test execution in SQL statements. Given the number of potential solutions to this problem, the proposed approach combines techniques Mutation Analysis for SQL with Evolutionary Computation to find a reduced data set, that be able to detect a large number of defects in SQL statements of a particular application. Based on a heuristic perspective, the proposal uses Genetic Algorithms (GA) to select tuples from a existing database (from production environment) trying to reduce it to a set of data relevant and effective. During the evolutionary process, Mutation Analysis is used to evaluate each set of test data selected by the AG. The results obtained from the experiments showed a good performance using meta-heuristic of Genetic Algorithms, and its variations. / Teste de Software é uma área da Engenharia de Software de fundamental importância para a garantia da qualidade do software. São atividades que envolvem tempo e custos elevados, mas que precisam ser realizadas durante todo o processo de construção de um software. Assim como em outra áreas da Engenharia de Software, existem problemas nas atividades de Teste de Software cuja solução não é trivial. Para esses problemas, têm sido exploradas várias técnicas de busca e otimização tentando encontrar uma solução ótima ou perto da ótima, dando origem às linhas de pesquisa Search-Based Software Engineering (SBSE) e Search-Based Software Testing (SBST). O presente trabalho está inserido neste contexto e tem como objetivo solucionar o problema de seleção de dados de teste para execução de testes em instruções SQL. Dada a quantidade de soluções possíveis para este problema, a abordagem proposta combina técnicas de Análise de Mutantes SQL com Computação Evolucionária para encontrar um conjunto de dados reduzido que seja capaz de detectar uma grande quantidade de defeitos em instruções SQL de uma determinada aplicação. Baseada em uma perspectiva heurística, a proposta utiliza Algoritmos Genéticos (AG) para selecionar tuplas de um banco de dados existente (de produção) tentando reduzi-lo em um conjunto de dados relevante e efetivo. Durante o processo evolucionário, a Análise de Mutantes é utilizada para avaliação de cada conjunto de dados de teste selecionado pelo AG. Os resultados obtidos com a realização dos experimentos revelaram um bom desempenho utilizando a metaheurística dos Algoritmos Genéticos e suas variações.

O teste de mutação é um critério de teste poderoso para detectar falhas e medir a eficácia de um conjunto de dados de teste. No entanto, é uma técnica de teste computacionalmente cara. O alto custo provém principalmente do esforço para gerar dados de teste adequados para matar os mutantes e pela existência de mutantes equivalentes. Nesse contexto, o objetivo desta tese é apresentar uma abordagem chamada de Reach, Infect and Propagation to Mutation Testing (RIPMuT) que visa gerar dados de teste e sugerir mutantes equivalentes. A abordagem é composta por dois módulos: (i) uma geração automatizada de dados de teste usando subida da encosta e um esquema de fitness de acordo com as condições de alcançabilidade, infeção e propagação (RIP); e (ii) um método para sugerir mutantes equivalentes com base na análise das condições RIP durante o processo de geração de dados de teste. Os experimentos foram conduzidos para avaliar a eficácia da abordagem RIP-MuT e um estudo comparativo com o algoritmo genético e testes aleatórios foi realizado. A abordagem RIP-MuT obteve um escore médio de mutação de 18,25 % maior que o AG e 35,93 % maior que o teste aleatório. O método proposto para detecção de mutantes equivalentes se mostrou viável para redução de custos relacionado a essa atividade, uma vez que obteve uma precisão de 75,05% na sugestão dos mutantes equivalentes. Portanto, os resultados indicam que a abordagem gera dados de teste adequados capazes de matar a maioria dos mutantes em programas C e, também auxilia a identificar mutantes equivalentes corretamente. / Mutation Testing is a powerful test criterion to detect faults and measure the effectiveness of a test data set. However, it is a computationally expensive testing technique. The high cost comes mainly from the effort to generate adequate test data to kill the mutants and by the existence of equivalent mutants. In this thesis, an approach called Reach, Infect and Propagation to Mutation Testing (RIP-MuT) is presented to generate test data and to suggest equivalent mutants. The approach is composed of two modules: (i) an automated test data generation using hill climbing and a fitness scheme according to Reach, Infect, and Propagate (RIP) conditions; and (ii) a method to suggest equivalent mutants based on the analyses of RIP conditions during the process of test data generation. The experiments were conducted to evaluate the effectiveness of the RIP-MuT approach and a comparative study with a genetic algorithm and random testing. The RIP-MuT approach achieved a mean mutation score of 18.25% higher than the GA and 35.93% higher than random testing. The proposed method for detection of equivalent mutants demonstrate to be feasible for cost reduction in this activity since it obtained a precision of 75.05% on suggesting equivalent mutants. Therefore, the results indicate that the approach produces effective test data able to strongly kill the majority of mutants on C programs, and also it can assist in suggesting equivalent mutants correctly.

Submitted by Jaqueline Silva (jtas29@gmail.com) on 2014-10-06T17:24:11Z No. of bitstreams: 2 Dissertação - André Assis Lôbo de Oliveira- 2013.pdf: 3915731 bytes, checksum: 2447fa437e5dca74e295727bd8fed4d1 (MD5) license_rdf: 23148 bytes, checksum: 9da0b6dfac957114c6a7714714b86306 (MD5) / Approved for entry into archive by Jaqueline Silva (jtas29@gmail.com) on 2014-10-06T19:18:43Z (GMT) No. of bitstreams: 2 Dissertação - André Assis Lôbo de Oliveira- 2013.pdf: 3915731 bytes, checksum: 2447fa437e5dca74e295727bd8fed4d1 (MD5) license_rdf: 23148 bytes, checksum: 9da0b6dfac957114c6a7714714b86306 (MD5) / Made available in DSpace on 2014-10-06T19:18:43Z (GMT). No. of bitstreams: 2 Dissertação - André Assis Lôbo de Oliveira- 2013.pdf: 3915731 bytes, checksum: 2447fa437e5dca74e295727bd8fed4d1 (MD5) license_rdf: 23148 bytes, checksum: 9da0b6dfac957114c6a7714714b86306 (MD5) Previous issue date: 2013-12-05 / Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES / Verification and Validation Activities (V&V) consume about 50% to 60% of the total cost of a software lifecycle. Among those activities, Software Testing technique is one which is mostly used during this process. One of the main problems related to detected in Software Testing is to find a set of tests (subset from input domain of the problem) which is effective to detect the remaining bugs in the software. The Search-Based Software Testing (SBST) approach uses metaheuristics to find low cost set of tests with a high effectiveness to detect bugs. From several existing test criteria, Mutation Testing is considered quite promising to reveal bugs, despite its high computational cost, due to the great quantity of mutant programs generated. Therefore, this dissertation addresses the problem of selecting mutant programs and test cases in Mutation Testing context. To this end, it is proposed a Coevolutionary Genetic Algorithm (CGA) and the concept of Genetic Effectiveness, implemented by Genetic Classification (GC) and new genetic operators adapted to the proposed representation. Furthermore, the Genetic Algorithm Coevolutionary with Controlled Genetic Classification (CGA􀀀CGCop) is proposed for improving the efficiency of CGA’s GC. The CGA is applied in three categories of benchmarks and compared to other five methods. The results show a better performance of the CGA in subsets selection with better mutation score, as well as improvement of CGA􀀀CGCop in use of GC. These results evidence the proposal approach with promising use in the context of Mutation Testing. / Atividades de Validação e Verificação (V&V) consomem cerca de 50% a 60% do custo total no ciclo de vida de um software. Dentre essas, o Teste de Software é uma das atividades mais empregadas. Um dos maiores problemas do Teste de Software é encontrar um conjunto de teste (subconjunto do domínio de entrada do problema) que seja eficaz em detectar os defeitos remanescentes no software. Neste contexto, a Search-Based Software Testing (SBST) é uma linha de pesquisa recente que vem propondo boas soluções, uma vez que utiliza-se de metaheurísticas para encontrar um conjunto de teste com baixo custo e grande eficácia na detecção de defeitos. Dentre os diversos critérios de teste existentes, o Teste de Mutação é bastante promissor na revelação de defeitos, entretanto apresenta um alto custo computacional em termos de aplicabilidade. Por isso, a pesquisa aborda o problema de seleção de programas mutantes e casos de teste no contexto do Teste de Mutação. Para tal, propõe o Algoritmo Genético Coevolucionário (AGC) que traz o conceito de Efetividade Genética, implementado pela Classificação Genética (CG) e por novos operadores genéticos adaptados à representação proposta. Além disso, propõe o Algoritmo Genético Coevolucionário com Classificação Genética Controlada (AGC 􀀀CGCop) para a melhoria da eficiência da CG do AGC. O algoritmo AGC é aplicado em três classes de benchmarks e comparado com outros cinco métodos. Os resultados demonstram um melhor desempenho do AGC na seleção de subconjuntos com melhor escore de mutação, bem como um aprimoramento do AGC􀀀CGCop no uso da CG. Tais resultados evidenciam a abordagem proposta com uso promissor no contexto do Teste de Mutação.