Mobile applications are growing in popularity and pose new problems in the area of software testing. In particular, mobile applications heavily depend upon user interactions and a dynamically changing environment of system events. In this thesis, we focus on user-driven events and use Q-learning, a reinforcement machine learning algorithm, to generate tests for Android applications under test (AUT). We implement a framework that automates the generation of GUI test cases by using our Q-learning approach and compare it to a uniform random (UR) implementation. A novel feature of our approach is that we generate user-driven event sequences through the GUI, without the source code or the model of the AUT. Hence, considerable amount of cost and time are saved by avoiding the need for model generation for generating the tests. Our results show that the systematic path exploration used by Q-learning results in higher average code coverage in comparison to the uniform random approach.

Bug Fixer is a web-based application that complements lectures with hands-on exercises that encourage students to think about the logic in programs. Bug Fixer presents students with code that has several bugs that they must fix. The process of fixing the bugs forces students to conceptually think about the code and reinforces their understanding of the logic behind algorithms. In this work, we conducted a study using Bug Fixer with undergraduate students in the CSCE1040 course at University of North Texas to evaluate whether the system increases their conceptual understanding of the algorithms and improves their Software Testing skills. Students participated in weekly activities to fix bugs in code. Most students enjoyed Bug Fixer and recommend the system for future use. Students typically reported a better understanding of the algorithms used in class. We observed a slight increase of passing grades for students who participated in our study compared to students in other sections of the course with the same instructor who did not participate in our study. The students who did not report a positive experience provide comments for future improvements that we plan to address in future work.

Regression testing of an evolving software system may involve significant challenges. While, there would be a requirement of maximising the probability of finding out if the latest changes to the system has broken some existing feature, it needs to be done as economically as possible. A particularly important class of software systems are API libraries. Such libraries would typically constitute a very important component of many software systems. High quality requirements make it imperative to continually optimise the internal implementation of such libraries without affecting the external interface. Therefore, it is preferred to guide the regression testing by some kind of formal specification of the library. The testing problem comprises of three parts: computation of test data, execution of test, and analysis of test results. Current research mostly focuses on the first part. The objective of test data computation is to maximise the probability of uncovering bugs, and to do it with as few test cases as possible. The problem of test data computation for regression testing is to select a subset of the original test suite running which would suffice to test for bugs probably inserted in the modifications done after the last round of testing. A variant of this problem is that of regression testing of API libraries. The regression testing of an API is usually done by making function calls in such a way that the sequence of function calls thus made suffices a test specification. The test specification in turn embodies some concept of completeness. In this thesis, we focus on the problem of test sequence computation for the regression testing of API libraries. At the heart of this method lies the creation of a state space model of the API library by reverse engineering it by executing the system, with guidance from an formal API specification. Once the state space graph is obtained, it is used to compute test sequences for satisfying some test specification. We analyse the theoretical complexity of the problem of test sequence computation and provide various heuristic algorithms for the same. State space explosion is a classical problem encountered whenever there is an attempt of creating a finite state model of a program. Our method also faces this limitation. We explore a simple and intuitive method of ameliorating this problem – by simply reducing the size of the state vector. We develop the theoretical insights into this method. Also, we present experimental results indicating the practical effectiveness of this method. Finally, we bring all this together into the design and implementation of a tool called Modest.

Introdução – A área de pesquisa de estimativa de esforço de software busca melhorar a acurácia das estimativas de projetos e atividades de software. Objetivo – Este trabalho descreve o desenvolvimento e uso de uma ferramenta web de coleta de dados gerados durante a execução da técnica de estimativa Planning Poker e a análise dos dados coletados para investigação do impacto da revisão de dados históricos de esforço. Método – Foram realizadas estimativas com e sem revisão, em experimentos com alunos de computação da Universidade Tecnológica Federal do Paraná, coletando os dados relacionados à tomada de decisão em uma ferramenta web. Após isso, foi analisado o impacto causado pelas revisões na acurácia da estimativa de esforço de software utilizando Planning Poker. Resultados Obtidos – Foi analisado o comportamento de 14 grupos de estimativas. Dentre esses times, 8 deles tiveram uma melhora na acurácia maior que 50% das estimativas analisadas. Em 3 deles, a soma das estimativas que tiveram melhora com as estimativas que permaneceram estáveis ultrapassou os 50%. Em apenas 3 deles, as estimativas tiveram redução de acurácia maior que 50%. Conclusões – A Revisão de Atividades Executadas, utilizando Planning Poker, melhorou a estimativa de esforço na maioria dos casos analisados, podendo ser um importante método para aprimorar o processo de desenvolvimento de software. / Abstract – Background – The software effort estimation research area aims to improve the accuracy of this estimation in software projects and activities. Aims – This study describes the development and usage of a web application tocollect data generated from the Planning Poker estimation process and the analysis of the collected data to investigate the impact of revising previous estimates when conducting similar estimates in a Planning Poker context. Method – Software activities were estimated by Universidade Tecnológica Federal do Paraná (UTFPR) computer students, using Planning Poker, with and without revising previous similar activities, storing data regarding the decision-making process. And the collected data was used to investigate the impact that revising similar executed activities have in the software effort estimates' accuracy.Obtained Results – The UTFPR computer students were divided into 14 groups. Eight of them showed accuracy increase in more than half of their estimates. Three of them had almost the same accuracy in more than half of their estimates. And only three of them had loss of accuracy in more than half of their estimates. Conclusion – Reviewing the similar executed software activities, when using Planning Poker, led to more accurate software estimates in most cases, and, because of that, can improve the software development process.

Introdução – A área de pesquisa de estimativa de esforço de software busca melhorar a acurácia das estimativas de projetos e atividades de software. Objetivo – Este trabalho descreve o desenvolvimento e uso de uma ferramenta web de coleta de dados gerados durante a execução da técnica de estimativa Planning Poker e a análise dos dados coletados para investigação do impacto da revisão de dados históricos de esforço. Método – Foram realizadas estimativas com e sem revisão, em experimentos com alunos de computação da Universidade Tecnológica Federal do Paraná, coletando os dados relacionados à tomada de decisão em uma ferramenta web. Após isso, foi analisado o impacto causado pelas revisões na acurácia da estimativa de esforço de software utilizando Planning Poker. Resultados Obtidos – Foi analisado o comportamento de 14 grupos de estimativas. Dentre esses times, 8 deles tiveram uma melhora na acurácia maior que 50% das estimativas analisadas. Em 3 deles, a soma das estimativas que tiveram melhora com as estimativas que permaneceram estáveis ultrapassou os 50%. Em apenas 3 deles, as estimativas tiveram redução de acurácia maior que 50%. Conclusões – A Revisão de Atividades Executadas, utilizando Planning Poker, melhorou a estimativa de esforço na maioria dos casos analisados, podendo ser um importante método para aprimorar o processo de desenvolvimento de software. / Abstract – Background – The software effort estimation research area aims to improve the accuracy of this estimation in software projects and activities. Aims – This study describes the development and usage of a web application tocollect data generated from the Planning Poker estimation process and the analysis of the collected data to investigate the impact of revising previous estimates when conducting similar estimates in a Planning Poker context. Method – Software activities were estimated by Universidade Tecnológica Federal do Paraná (UTFPR) computer students, using Planning Poker, with and without revising previous similar activities, storing data regarding the decision-making process. And the collected data was used to investigate the impact that revising similar executed activities have in the software effort estimates' accuracy.Obtained Results – The UTFPR computer students were divided into 14 groups. Eight of them showed accuracy increase in more than half of their estimates. Three of them had almost the same accuracy in more than half of their estimates. And only three of them had loss of accuracy in more than half of their estimates. Conclusion – Reviewing the similar executed software activities, when using Planning Poker, led to more accurate software estimates in most cases, and, because of that, can improve the software development process.

This thesis examines integration testing of object-oriented software. The process of integrating and testing procedural programs is reviewed as foundation for testing object-oriented software. The complexity of object-oriented software is examined. The relationship of integration testing and the software development life cycle is presented. Scenarios are discussed which account for the introduction of defects into the software. The Unified Modeling Language (UML) is chosen for representing pre-implementation and post-implementation models of the software. A demonstration of the technique of using post-implementation models representing the logical and physical views as an aid in integration and system testing of the software is presented. The use of UML diagrams developed from the software is suggested as a technique for integration testing of object-oriented software. The need for automating the data collection and model building is recognized. The technique is integrated into the Revised Spiral Model for Object-Oriented Software Development developed by du Plessis and van der Walt. / Computing / D.Phil. (Computer Science)

This thesis examines integration testing of object-oriented software. The process of integrating and testing procedural programs is reviewed as foundation for testing object-oriented software. The complexity of object-oriented software is examined. The relationship of integration testing and the software development life cycle is presented. Scenarios are discussed which account for the introduction of defects into the software. The Unified Modeling Language (UML) is chosen for representing pre-implementation and post-implementation models of the software. A demonstration of the technique of using post-implementation models representing the logical and physical views as an aid in integration and system testing of the software is presented. The use of UML diagrams developed from the software is suggested as a technique for integration testing of object-oriented software. The need for automating the data collection and model building is recognized. The technique is integrated into the Revised Spiral Model for Object-Oriented Software Development developed by du Plessis and van der Walt. / Computing / D.Phil. (Computer Science)

Test Driven Development (TDD) é uma prática ágil que ganhou popularidade ao ser definida como parte fundamental na eXtreme Programming (XP). Essa prática determina que os testes devem ser escritos antes da implementação do código. TDD e seus efeitos têm sido amplamente estudados e comparados com a prática Test Last Development (TLD) em diversos trabalhos. Entretanto, poucos estudos abordam TDD no desenvolvimento de Web Services (WS), devido à complexidade em testar as dependências entre os componentes distribuídos e as particularidades da Service Oriented Architecture (SOA). Este trabalho tem por objetivo definir e validar uma abordagem para o desenvolvimento de WS baseada na prática de TDD, denominada WS-TDD. Essa abordagem guia os desenvolvedores no uso de TDD durante o desenvolvimento de WS, sugerindo ferramentas e técnicas para lidar com as dependências e as particularidades de SOA, com foco na criação dos testes unitários e integrados automatizados na linguagem Java. No intuito de definir e validar a abordagem proposta, quatro métodos de pesquisa foram executados: (i) questionário presencial; (ii) experimento; (iii) entrevista presencial com cada participante do experimento e (iv) triangulação dos resultados com as pessoas que participaram nos três métodos anteriores. De acordo com os resultados obtidos, a WS-TDD mostrou-se mais eficiente quando comparada a TLD, aumentando a qualidade interna do software e a produtividade dos desenvolvedores. No entanto, a qualidade externa do software diminuiu, apresentando um maior número de defeitos quando comparada a TLD. Por fim, é importante destacar que a abordagem proposta surge como uma alternativa simples e prática para a adoção de TDD no desenvolvimento de WS, trazendo benefícios a qualidade interna e contribuindo para aumentar a produtividade dos desenvolvedores. Porém, a qualidade externa do software diminuiu ao utilizar a WS-TDD. / Test Driven Development (TDD) is an agile practice that gained popularity when defined as a fundamental part in eXtreme Programming (XP). This practice determines that the tests should be written before implementing the code. TDD and its effects have been widely studied and compared with the Test Last Development (TLD) in several studies. However, few studies address TDD practice in the development of Web Services (WS), due to the complexity of testing the dependencies among distributed components and the specific characteristics of Service Oriented Architecture (SOA). This study aims to define and validate an approach to develop WS based on the practice of TDD, called WS-TDD. This approach guides developers to use TDD to develop WS, suggesting tools and techniques to deal with SOA particularities and dependencies, focusing on the creation of the unitary and integrated automated tests in Java. In order to define and validate the proposed approach, four research methods have been carried out: (i) questionnaire; (ii) practical experiment; (iii) personal interview with each participant in the experiment and (iv) triangulation of the results with the people who participated in the three previous methods. According to the obtained results, WS-TDD was more efficient compared to TLD, increasing internal software quality and developer productivity. However, the external software quality has decreased due to a greater number of defects compared to the TLD approach. Finally, it is important to highlight that the proposed approach is a simple and practical alternative for the adoption of TDD in the development of WS, bringing benefits to internal quality and contributing to increase the developers’ productivity. However, the external software quality has decreased when using WS-TDD.

Com a necessidade cada vez maior de agilizar a entrega de novos desenvolvimentos ao cliente e de diminuir o tempo de desenvolvimento das aplicações, a priorização de casos de teste possibilita a detecção das falhas presentes na aplicação mais rapidamente por meio da ordenação dos casos de teste a serem executados. E, com isso, possibilita também que a correção destas falhas inicie o mais brevemente possível. Entretanto, quando os casos de teste a serem priorizados são testes automatizados de sistema, critérios tradicionais utilizados na literatura como cobertura de código ou modelos do sistema deixam de ser interessantes, dada a característica inerente deste tipo de teste na qual a organização e a modelagem adotadas são ignoradas por se tratarem de testes de caixa preta. Considerando a hipótese de que casos de teste automatizados grandes testam mais partes da aplicação e que casos de teste similares podem estar testando a mesma área da aplicação, parece válido crer que a execução dos casos de teste de sistema priorizando os testes mais complexos pode alcançar resultados positivos quando comparada à execução não ordenada dos casos de teste. É neste cenário que este trabalho propõe o uso dos grafos de chamadas dos próprios casos de teste como critério para priorização destes, priorizando assim a execução dos casos de teste com a maior quantidade de nós no seu grafo. A abordagem proposta neste trabalho mostrou, por meio de dois estudos de caso, ser capaz de melhorar a taxa de detecção de falhas em relação à execução não ordenada dos casos de teste. Além disso, a abordagem proposta obteve resultados semelhantes as abordagens tradicionais de priorização utilizando cobertura de código da aplicação. / With the increasing need to streamline the delivery of new developments to the customer and reduce application development time, test case prioritization allows a quicker detection of faults present in the application through the ordering of test cases to be executed. Besides that, a quicker detection enables also the correction of these faults to start as soon as possible. However, when the test cases to be prioritized are automated system tests, traditional criteria used in the literature like code coverage or system models become uninteresting, given that this type of test case, classified as black box test, ignores how the application was coded or modeled. Considering the hypothesis that bigger automated test cases verify more parts of the application and that similar test cases may be testing the same application areas, it seems valid to believe that giving a higher priority to more complex test cases to be executed first can accomplish positive results when compared to the unordered execution of test cases. It is on this scenario that this project studies the usage of call graphs from test cases as the criterion to prioritize them, increasing the priority of the execution of test cases with the higher number of nodes on the graph. The approach proposed in this document showed through two case studies that it is capable of improving fault detection rate compared to unordered test cases. Furthermore, the proposed approach achieved similar results when compared to a traditional prioritization approach using code coverage of the application.

Test Driven Development (TDD) é uma prática ágil que ganhou popularidade ao ser definida como parte fundamental na eXtreme Programming (XP). Essa prática determina que os testes devem ser escritos antes da implementação do código. TDD e seus efeitos têm sido amplamente estudados e comparados com a prática Test Last Development (TLD) em diversos trabalhos. Entretanto, poucos estudos abordam TDD no desenvolvimento de Web Services (WS), devido à complexidade em testar as dependências entre os componentes distribuídos e as particularidades da Service Oriented Architecture (SOA). Este trabalho tem por objetivo definir e validar uma abordagem para o desenvolvimento de WS baseada na prática de TDD, denominada WS-TDD. Essa abordagem guia os desenvolvedores no uso de TDD durante o desenvolvimento de WS, sugerindo ferramentas e técnicas para lidar com as dependências e as particularidades de SOA, com foco na criação dos testes unitários e integrados automatizados na linguagem Java. No intuito de definir e validar a abordagem proposta, quatro métodos de pesquisa foram executados: (i) questionário presencial; (ii) experimento; (iii) entrevista presencial com cada participante do experimento e (iv) triangulação dos resultados com as pessoas que participaram nos três métodos anteriores. De acordo com os resultados obtidos, a WS-TDD mostrou-se mais eficiente quando comparada a TLD, aumentando a qualidade interna do software e a produtividade dos desenvolvedores. No entanto, a qualidade externa do software diminuiu, apresentando um maior número de defeitos quando comparada a TLD. Por fim, é importante destacar que a abordagem proposta surge como uma alternativa simples e prática para a adoção de TDD no desenvolvimento de WS, trazendo benefícios a qualidade interna e contribuindo para aumentar a produtividade dos desenvolvedores. Porém, a qualidade externa do software diminuiu ao utilizar a WS-TDD. / Test Driven Development (TDD) is an agile practice that gained popularity when defined as a fundamental part in eXtreme Programming (XP). This practice determines that the tests should be written before implementing the code. TDD and its effects have been widely studied and compared with the Test Last Development (TLD) in several studies. However, few studies address TDD practice in the development of Web Services (WS), due to the complexity of testing the dependencies among distributed components and the specific characteristics of Service Oriented Architecture (SOA). This study aims to define and validate an approach to develop WS based on the practice of TDD, called WS-TDD. This approach guides developers to use TDD to develop WS, suggesting tools and techniques to deal with SOA particularities and dependencies, focusing on the creation of the unitary and integrated automated tests in Java. In order to define and validate the proposed approach, four research methods have been carried out: (i) questionnaire; (ii) practical experiment; (iii) personal interview with each participant in the experiment and (iv) triangulation of the results with the people who participated in the three previous methods. According to the obtained results, WS-TDD was more efficient compared to TLD, increasing internal software quality and developer productivity. However, the external software quality has decreased due to a greater number of defects compared to the TLD approach. Finally, it is important to highlight that the proposed approach is a simple and practical alternative for the adoption of TDD in the development of WS, bringing benefits to internal quality and contributing to increase the developers’ productivity. However, the external software quality has decreased when using WS-TDD.