Integral part of all systematically managed software development or maintenance projects is emphasis on continuous quality of all project activities. Because final quality of project deliverables (new or enhanced applications, preconfigured solutions as ie. SAP) is a very big influencer of project success and therefore also important influencer of long-term relationship between customer and contractor(s), this document focuses on ways how to proactively prevent from mistakes (within the whole software development lifecycle) and on techniques helping to establish better quality control of important deliverables through systematic approach, high quality tools and suitable metrics in software testing discipline. This document gradually exposes typical project areas where it is necessary to keep control on quality of project members' outputs, perceives testing in context of typical project consequences, offers practical recommendations in testing methodology, tools as well as widely tested technologies, and explains trends and risks in testing domain. The goal of this document is not only to document a wide range of possibilities given by frequently used testing techniques or tools but also to offer a practical guidance in deployment of test discipline. This document was written by comparing author's professional experience in software quality management with knowledge gathered by reading information sources attached to this document. This document consists of concrete conclusions of this comparison.

The rapid advancement of web and mobile application technologies has recently posed numerous challenges to the Software Engineering community, including how to cost-effectively test applications that have complex event spaces. Many software testing techniques attempt to cost-effectively improve the quality of such software. This dissertation primarily focuses on that of hybrid test suite prioritization. The techniques utilize two or more criteria to perform test suite prioritization as it is often insufficient to use only a single criterion. The dissertation consists of the following contributions: (1) a weighted test suite prioritization technique that employs the distance between criteria as a weighting factor, (2) a coarse-to-fine grained test suite prioritization technique that uses a multilevel approach to increase the granularity of the criteria at each subsequent iteration, (3) the Caret-HM tool for Android user session-based testing that allows testers to record, replay, and create heat maps from user interactions with Android applications via a web browser, and (4) Android user session-based test suite prioritization techniques that utilize heuristics developed from user sessions created by Caret-HM. Each of the chapters empirically evaluate the respective techniques. The proposed techniques generally show improved or equally good performance when compared to the baselines, depending on an application under test. Further, this dissertation provides guidance to testers as it relates to the use of the proposed hybrid techniques.

O teste de mutação é um critério de teste poderoso para detectar falhas e medir a eficácia de um conjunto de dados de teste. No entanto, é uma técnica de teste computacionalmente cara. O alto custo provém principalmente do esforço para gerar dados de teste adequados para matar os mutantes e pela existência de mutantes equivalentes. Nesse contexto, o objetivo desta tese é apresentar uma abordagem chamada de Reach, Infect and Propagation to Mutation Testing (RIPMuT) que visa gerar dados de teste e sugerir mutantes equivalentes. A abordagem é composta por dois módulos: (i) uma geração automatizada de dados de teste usando subida da encosta e um esquema de fitness de acordo com as condições de alcançabilidade, infeção e propagação (RIP); e (ii) um método para sugerir mutantes equivalentes com base na análise das condições RIP durante o processo de geração de dados de teste. Os experimentos foram conduzidos para avaliar a eficácia da abordagem RIP-MuT e um estudo comparativo com o algoritmo genético e testes aleatórios foi realizado. A abordagem RIP-MuT obteve um escore médio de mutação de 18,25 % maior que o AG e 35,93 % maior que o teste aleatório. O método proposto para detecção de mutantes equivalentes se mostrou viável para redução de custos relacionado a essa atividade, uma vez que obteve uma precisão de 75,05% na sugestão dos mutantes equivalentes. Portanto, os resultados indicam que a abordagem gera dados de teste adequados capazes de matar a maioria dos mutantes em programas C e, também auxilia a identificar mutantes equivalentes corretamente. / Mutation Testing is a powerful test criterion to detect faults and measure the effectiveness of a test data set. However, it is a computationally expensive testing technique. The high cost comes mainly from the effort to generate adequate test data to kill the mutants and by the existence of equivalent mutants. In this thesis, an approach called Reach, Infect and Propagation to Mutation Testing (RIP-MuT) is presented to generate test data and to suggest equivalent mutants. The approach is composed of two modules: (i) an automated test data generation using hill climbing and a fitness scheme according to Reach, Infect, and Propagate (RIP) conditions; and (ii) a method to suggest equivalent mutants based on the analyses of RIP conditions during the process of test data generation. The experiments were conducted to evaluate the effectiveness of the RIP-MuT approach and a comparative study with a genetic algorithm and random testing. The RIP-MuT approach achieved a mean mutation score of 18.25% higher than the GA and 35.93% higher than random testing. The proposed method for detection of equivalent mutants demonstrate to be feasible for cost reduction in this activity since it obtained a precision of 75.05% on suggesting equivalent mutants. Therefore, the results indicate that the approach produces effective test data able to strongly kill the majority of mutants on C programs, and also it can assist in suggesting equivalent mutants correctly.

Submitted by Marlene Santos (marlene.bc.ufg@gmail.com) on 2014-10-15T17:49:53Z No. of bitstreams: 2 Dissertacao - Ana Claudia Bastos Loureiro Monção - 2013.pdf: 4213405 bytes, checksum: 3bbe190ae0f4a45a2f8b4e71026f5d2e (MD5) license_rdf: 23148 bytes, checksum: 9da0b6dfac957114c6a7714714b86306 (MD5) / Approved for entry into archive by Jaqueline Silva (jtas29@gmail.com) on 2014-10-16T17:59:00Z (GMT) No. of bitstreams: 2 Dissertacao - Ana Claudia Bastos Loureiro Monção - 2013.pdf: 4213405 bytes, checksum: 3bbe190ae0f4a45a2f8b4e71026f5d2e (MD5) license_rdf: 23148 bytes, checksum: 9da0b6dfac957114c6a7714714b86306 (MD5) / Made available in DSpace on 2014-10-16T17:59:00Z (GMT). No. of bitstreams: 2 Dissertacao - Ana Claudia Bastos Loureiro Monção - 2013.pdf: 4213405 bytes, checksum: 3bbe190ae0f4a45a2f8b4e71026f5d2e (MD5) license_rdf: 23148 bytes, checksum: 9da0b6dfac957114c6a7714714b86306 (MD5) Previous issue date: 2013-08-02 / Software Testing is an important area of Software Engineering to ensuring the software quality. It consists of activities that involve long time and high costs, but need to be made throughout the process of building software. As in other areas of software engineering, there are problems in the activities of Software Testing whose solution is not trivial. For these problems, several techniques of optimization and search have been explored trying to find an optimal solution or near optimal, giving rise to lines of research textit Search-Based Software Engineering (SBSE) and textit Search-Based Software Testing (SBST). This work is part of this context and aims to solve the problem of selecting test data for test execution in SQL statements. Given the number of potential solutions to this problem, the proposed approach combines techniques Mutation Analysis for SQL with Evolutionary Computation to find a reduced data set, that be able to detect a large number of defects in SQL statements of a particular application. Based on a heuristic perspective, the proposal uses Genetic Algorithms (GA) to select tuples from a existing database (from production environment) trying to reduce it to a set of data relevant and effective. During the evolutionary process, Mutation Analysis is used to evaluate each set of test data selected by the AG. The results obtained from the experiments showed a good performance using meta-heuristic of Genetic Algorithms, and its variations. / Teste de Software é uma área da Engenharia de Software de fundamental importância para a garantia da qualidade do software. São atividades que envolvem tempo e custos elevados, mas que precisam ser realizadas durante todo o processo de construção de um software. Assim como em outra áreas da Engenharia de Software, existem problemas nas atividades de Teste de Software cuja solução não é trivial. Para esses problemas, têm sido exploradas várias técnicas de busca e otimização tentando encontrar uma solução ótima ou perto da ótima, dando origem às linhas de pesquisa Search-Based Software Engineering (SBSE) e Search-Based Software Testing (SBST). O presente trabalho está inserido neste contexto e tem como objetivo solucionar o problema de seleção de dados de teste para execução de testes em instruções SQL. Dada a quantidade de soluções possíveis para este problema, a abordagem proposta combina técnicas de Análise de Mutantes SQL com Computação Evolucionária para encontrar um conjunto de dados reduzido que seja capaz de detectar uma grande quantidade de defeitos em instruções SQL de uma determinada aplicação. Baseada em uma perspectiva heurística, a proposta utiliza Algoritmos Genéticos (AG) para selecionar tuplas de um banco de dados existente (de produção) tentando reduzi-lo em um conjunto de dados relevante e efetivo. Durante o processo evolucionário, a Análise de Mutantes é utilizada para avaliação de cada conjunto de dados de teste selecionado pelo AG. Os resultados obtidos com a realização dos experimentos revelaram um bom desempenho utilizando a metaheurística dos Algoritmos Genéticos e suas variações.

O teste de mutação é um critério de teste poderoso para detectar falhas e medir a eficácia de um conjunto de dados de teste. No entanto, é uma técnica de teste computacionalmente cara. O alto custo provém principalmente do esforço para gerar dados de teste adequados para matar os mutantes e pela existência de mutantes equivalentes. Nesse contexto, o objetivo desta tese é apresentar uma abordagem chamada de Reach, Infect and Propagation to Mutation Testing (RIPMuT) que visa gerar dados de teste e sugerir mutantes equivalentes. A abordagem é composta por dois módulos: (i) uma geração automatizada de dados de teste usando subida da encosta e um esquema de fitness de acordo com as condições de alcançabilidade, infeção e propagação (RIP); e (ii) um método para sugerir mutantes equivalentes com base na análise das condições RIP durante o processo de geração de dados de teste. Os experimentos foram conduzidos para avaliar a eficácia da abordagem RIP-MuT e um estudo comparativo com o algoritmo genético e testes aleatórios foi realizado. A abordagem RIP-MuT obteve um escore médio de mutação de 18,25 % maior que o AG e 35,93 % maior que o teste aleatório. O método proposto para detecção de mutantes equivalentes se mostrou viável para redução de custos relacionado a essa atividade, uma vez que obteve uma precisão de 75,05% na sugestão dos mutantes equivalentes. Portanto, os resultados indicam que a abordagem gera dados de teste adequados capazes de matar a maioria dos mutantes em programas C e, também auxilia a identificar mutantes equivalentes corretamente. / Mutation Testing is a powerful test criterion to detect faults and measure the effectiveness of a test data set. However, it is a computationally expensive testing technique. The high cost comes mainly from the effort to generate adequate test data to kill the mutants and by the existence of equivalent mutants. In this thesis, an approach called Reach, Infect and Propagation to Mutation Testing (RIP-MuT) is presented to generate test data and to suggest equivalent mutants. The approach is composed of two modules: (i) an automated test data generation using hill climbing and a fitness scheme according to Reach, Infect, and Propagate (RIP) conditions; and (ii) a method to suggest equivalent mutants based on the analyses of RIP conditions during the process of test data generation. The experiments were conducted to evaluate the effectiveness of the RIP-MuT approach and a comparative study with a genetic algorithm and random testing. The RIP-MuT approach achieved a mean mutation score of 18.25% higher than the GA and 35.93% higher than random testing. The proposed method for detection of equivalent mutants demonstrate to be feasible for cost reduction in this activity since it obtained a precision of 75.05% on suggesting equivalent mutants. Therefore, the results indicate that the approach produces effective test data able to strongly kill the majority of mutants on C programs, and also it can assist in suggesting equivalent mutants correctly.

Submitted by Natalia de Souza Gonçalves (natalia.goncalves@ufpe.br) on 2016-09-28T12:23:59Z No. of bitstreams: 2 license_rdf: 1232 bytes, checksum: 66e71c371cc565284e70f40736c94386 (MD5) Dissertacao_Cleydiane_FINAL_Imprimir.pdf: 3971527 bytes, checksum: f06a014d16a761f030bbfccee75d176f (MD5) / Made available in DSpace on 2016-09-28T12:23:59Z (GMT). No. of bitstreams: 2 license_rdf: 1232 bytes, checksum: 66e71c371cc565284e70f40736c94386 (MD5) Dissertacao_Cleydiane_FINAL_Imprimir.pdf: 3971527 bytes, checksum: f06a014d16a761f030bbfccee75d176f (MD5) Previous issue date: 2016-02-19 / Dívida Técnica (DT) está relacionada a tarefas que devem ser executadas e são acumuladas ao longo de um projeto para serem realizadas posteriormente. A falta de planejamento de tempo e recursos em um projeto, dentre outros problemas, propiciam seu surgimento. DTs não solucionadas tendem a criar dependências e aumentar o grau de complexidade para correção, resultando em maior esforço, custo e retrabalho em um projeto. A sua identificação e gerenciamento ao longo de um projeto é essencial para minimizar seu impacto e consequências negativas. Estudos apontam teste de software como uma das principais áreas impactadas por DTs em projetos de software. Nesta dissertação foram mapeadas, a partir da literatura técnica, 22 possíveis DTs relacionadas ao processo de teste de software, assim como suas causas e indicadores. Elas foram avaliadas por meio de um survey com profissionais da área de teste de software. Como resultados do survey, são apresentados os níveis de concordância obtidos para cada DT, suas causas e indicadores sugeridos. Após essa validação da concordância quanto às DTs, foi identificada a necessidade de vincular soluções às causas de cada DT, para isto foi realizada uma nova revisão de literatura. A partir dos dados das DTs, com suas causas, indicadores e, agora, soluções, foi elaborado um mapa, com objetivo de apoiar profissionais da área de teste de software na gestão de DTs que podem ocorrer no decorrer da execução do processo de teste de software. Após a elaboração do mapa de gestão de dívida técnica, o mesmo foi avaliado a partir de um questionário (baseado na técnica de revisão em pares), por profissionais experientes em teste de software (líderes e consultores). Como resultado desta avaliação, são apresentadas as opiniões e considerações destes profissionais sobre a versão inicial do mapa elaborado. Por fim, é apresentado o resultado final do mapa, com as devidas alterações baseadas nas respostas obtidas no questionário de avaliação do mapa. / Technical Debit (TD) is related to the tasks which have to be executed but are accumulated during the development of a project to be implemented later. The lack of time and resource planning on a project, among other problems, facilitate its emergence. Unsolved TDs tend to create dependencies and increase the degree of complexity for corrections, resulting in bigger efforts, costs and rework on a project. Identifying and managing those issues are essential to minimize the impacts and negative consequences. Studies point out software testing as one of the most affected areas by TDs on a software project. At this dissertation, 22 (twenty-two) possible TDs related to testing software process were mapped from the technical lecture, along with its causes and indicators. They were evaluated through surveys with testing software experts. As result of the survey, it is presented the levels of accordance for each TD, also the suggested causes and indicators. After the accordance validation of the TDs, it was identified a demand to bind solutions to the causes of every TD, for this a new study was made. From the data of the TDs, along with its causes, indicators and solutions, a map was produced in order to support testing software professionals managing TDs that may appear during the testing software process execution. After producing the technical debit management map, the same was submitted for an evaluation, using a questionnaire (based on peer review technique), to testing software experts (leaders and consultants). As result of these evaluations, it is presented the opinions and remarks of those professionals about the final version of the produced map. Finally, it is presented the final result of the map, with the modifications based on the answers obtained through the map evaluation questionnaire.

Context: Software testing is the process of assessing quality of a software product to determine whether it matches with the existing requirements of the customer or not. Software testing is one of the “Verification and Validation,” or V&V, software practices. The two basic techniques of software testing are Black-box testing and White box testing. Black-box testing focuses solely on the outputs generated in response to the inputs supplied neglecting the internal components of the software. Whereas, White-box testing focuses on the internal mechanism of the software of any application. To explore the feasibility of black-box and white-box testing under a given set of conditions, a proper test automation framework needs to be deployed. Automation is deployed in order to reduce the manual effort and to perform testing continuously, thereby increasing the quality of the product.   Objectives: In this research, cloud hosted application is automated using TestComplete tool. The objective of this thesis is to verify the functionality of Cloud application known as Test data library or Test Report Analyzer through automation and to measure the impact of the automation on release cycles of the organization.   Methods: Here automation is implemented using scrum methodology which is an agile development software process. Using scrum methodology, the product with working software can be delivered to the customers incrementally and empirically with updating functionalities in it. Test data library or Test Report Analyzer functionality of Cloud application is verified deploying testing device thereby the test cases can be analyzed thereby analyzing the pass or failed test cases.   Results: Automation of test report analyzer functionality of cloud hosted application is made using TestComplete and impact of automation on release cycles is reduced. Using automation, nearly 24% of change in release cycles can be observed thereby reducing the manual effort and increasing the quality of delivery.   Conclusion: Automation of a cloud hosted application provides no manual effort thereby utilization of time can be made effectively and application can be tested continuously increasing the efficiency and the quality of an application. / AUTOMATION OF A CLOUD HOSTED APPLICATION

Many challenges in software quality can be tackled with dynamic analysis. However, these techniques are often limited in their efficiency or scalability as they are often applied uniformly to an entire program. In this thesis, we show that dynamic program analysis can be made significantly more efficient and scalable by first performing a static data flow analysis so that the dynamic analysis can be selectively applied only to important parts of the program. We apply this general principle to the design and implementation of two different systems, one for runtime security policy enforcement and the other for software test input generation. For runtime security policy enforcement, we enforce user-defined policies using a dynamic data flow analysis that is more general and flexible than previous systems. Our system uses the user-defined policy to drive a static data flow analysis that identifies and instruments only the statements that may be involved in a security vulnerability, often eliminating the need to track most objects and greatly reducing the overhead. For taint analysis on a set of five server programs, the slowdown is only 0.65%, two orders of magnitude lower than previous taint tracking systems. Our system also has negligible overhead on file disclosure vulnerabilities, a problem that taint tracking cannot handle. For software test case generation, we introduce the idea of targeted testing, which focuses testing effort on select parts of the program instead of treating all program paths equally. Our “Bullseye” system uses a static analysis performed with respect to user-defined “interesting points” to steer the search down certain paths, thereby finding bugs faster. We also introduce a compiler transformation that allows symbolic execution to automatically perform boundary condition testing, revealing bugs that could be missed even if the correct path is tested. For our set of 9 benchmarks, Bullseye finds bugs an average of 2.5× faster than a conventional depth-first search and finds numerous bugs that DFS could not. In addition, our automated boundary condition testing transformation allows both Bullseye and depth-first search to find numerous bugs that they could not find before, even when all paths were explored. / text

Efficiency as a term in software testing is, in the research community, a term that is not so well defined. In the industry, and specifically the test tool industry, it has become a sales pitch without meaning. GUI testing in its manual form is a time consuming task, which can be thought of as repetitive and tedious by testers. Using human testers to perform a task, where focus is hard to keep, often ends in defects going unnoticed. The purpose of this thesis is to collect knowledge on the area efficiency in software testing, but focusing more on efficiency in GUI testing in order to keep the scope focused. Part of the purpose is also to test the hypothesis that automated GUI testing is more efficient than traditional, manual GUI testing. In order to reach the purpose, the choice fell to use case study research as the main research method. Through the case study, a theoretical study was performed to gain knowledge on the subject. To gain data used for an analysis in the case study, the choice fell on using a semi-experimental research approach where one automated GUI testing technique called Capture & Replay was tested against a more traditional approach towards GUI testing. The results obtained throughout the case study gives a definition on efficiency in software testing, as well as three measurements on efficiency, those being defect detection, repeatability of test cases, and time spent with human interaction. The result also includes the findings from the semi-experimental research approach where the testing tools Squish, and TestComplete, where used beside a manual testing approach. The main conclusion deducted in this work is that an automated approach towards GUI testing can become more efficient than a manual approach, in the long run. This is when efficiency is determined on the points of defect detection, repeatability, and time.

Introductory programming courses compose the core of several undergraduate degree programs, since programming is a crucial technical skill for professionals in many STEM areas. Aiming to address students learning difficulties in these courses, instructors can adopt different teaching approaches, since there are several varying aspects in programming education (e.g. programming languages and paradigms, development practices, platforms, supporting tools etc). In particular, the teaching approach that consists of integrating software testing into this context has been prominent in the area, since it may lead students to think more critically while working on programming assignments. Even so, this teaching approach can also present significant challenges, such as the students reluctance to conduct testing practices. In this sense, experimental studies have the role to provide evidence about learning outcomes, considering different teaching approaches and contexts. However, studies in the area of programming education often present a lack of theoretical basis, i.e. are not built upon established theories, models and frameworks. In other words, the varying aspects (or variables) used to investigate teaching approaches are not properly characterized in the studies, what leads to difficulties to interpret the obtained results and build knowledge in the area. As a consequence, instructors are prevented from having reliable evidence to make informed choices of teaching approaches used in the classroom. Considering this scenario, we propose in this PhD thesis the use of domain-specific models to support researchers in scoping and designing experiments on programming education. More specifically, we explored the domain of the software testing integration teaching approach, by creating a framework for experimental studies on the integration of software testing into programming education. The framework provides a basic structure of experimental studies in this domain, composed by models of variables related to this teaching approach. We also conducted experiments on the same domain and demonstrated their instantiation into the framework. We intend to support researchers and instructors in the scoping and planning of experimental studies in the educational scenario, specially those aimed at evaluating the integration of software testing into programming courses. / Disciplinas introdutórias de programação compõem o núcleo de diversos cursos de graduação, visto que se trata de uma habilidade crucial para profissionais em muitas áreas de ciências exatas. Buscando lidar com as dificuldades de aprendizagem dos alunos nessas disciplinas, os professores podem adotar diferentes abordagens de ensino, uma vez que há muitas variantes no ensino de programação (como linguagens e paradigmas de programação, práticas de desenvolvimento, plataformas, ferramentas de apoio, etc). Em particular, a abordagem de ensino que consiste em integrar teste de software nesse contexto tem se destacado na área, pois pode levar os alunos a pensarem de modo mais crítico enquanto resolvem atividades práticas de programação. Mesmo assim, essa abordagem de ensino também pode apresentar desafios significativos, como a resistência dos alunos para conduzir práticas de teste. Nesse sentido, estudos experimentais têm o papel de fornecer evidência acerca de resultados em termos de aprendizagem, considerando diferentes abordagens de ensino e contextos. Porém, estudos na área de ensino de programação muitas vezes apresentam falta de fundamentação teórica, ou seja, não são construídos a partir de teorias, modelos e frameworks estabelecidos na área. Isso significa que os aspectos (ou variáveis) utilizados para investigar as abordagens de ensino não são adequadamente caracterizados nos estudos, o que leva a dificuldades em interpretar os resultados obtidos e construir conhecimento na área. Como consequência, os professores são impedidos de ter evidências confiáveis para fazer escolhas informadas nas abordagens de ensino utilizadas em sala de aula. Considerando esse cenário, este trabalho de doutorado propõe o uso de modelos de domínio para apoiar pesquisadores ao definir e projetar experimentos no ensino de programação. Mais especificamente, o domínio da abordagem de integração de teste de software foi explorado neste trabalho, com a criação de um framework para estudos experimentais sobre a integração de teste de software no ensino de programação. O framework provê uma estrutura básica de estudos experimentais nesse domínio, sendo composto por modelos de variáveis relacionadas a essa abordagem de ensino. Neste trabalho também foram conduzidos experimentos de acordo com a estrutura do framework. A meta deste trabalho é apoiar pesquisadores e professores ao definir e planejar estudos no cenário educacional, em especial os focados em avaliar a integração de teste de software em disciplinas de programação.