Flaky tests are non-deterministic tests that both pass and fail when no new code changes have been introduced to the code under test. This is a widespread problem for the Continuous Integration community where the underlying idea is that code is only integrated if all test cases pass. Debugging and eradicating flaky tests is hard and time-consuming and if ignored may introduce bugs. Due to flaky tests, Continuous Integration systems are at risk of decreasing either productivity or quality, properties it claims to provide. Prior research suggests a link between 4 root causes for flaky tests and 5 different test smells, i.e. anti-code patterns. Unfortunately, difficulties in reproducing flaky tests found in the study make this connection ambiguous. This thesis intends to validate this relationship by re-implementing the test smells in a static analyser named FlakyHoover and detecting them on a new dataset consisting of order-dependent and non-order-dependent flaky tests. Analysis of the test smell distribution over order and non-order-dependent flaky tests makes it possible to determine the relationship between test smells and root causes for flaky tests. The findings suggest that there may exist a correlation between 3/5 test smells and certain types of root causes for flaky tests. However further research is required to determine the studied test smells relation to flaky tests.

Quality assurance is becoming increasingly important to implement in expanding development projects. The components being built must be tested regularly to ensure the functionality of the whole product. Leeroy applies scrum as a development method along with manual GUI tests that only are performed when all components belonging to each sprint are completed. This leads to shortage in time for a tester to apply the assigned test scenarios. Testers are therefore subjected to stress in their work as testing requires high accuracy and speed in order to effectively ensure that the components end up in a production environment. The purpose of this study is to streamline current test processes by replacing manual GUI tests with automated GUI regression tests using BDD principles. Specification of the requirements was carried out using data collection methods that were chosen for this study. The specifications were used both to evaluate the construction as well as the results obtained. The results show that it is important that both the tester and the rest of the agile team cooperate during each sprint, as the components are developed incrementally and tested regularly. The results show even the importance of prioritizing the test scenarios; this is done to ensure the efficiency of the workflow by first choosing the most important components to test. The tested aspects in this study have also proven that both frameworks Cucumber and Puppeteer have the potential to contribute to a smooth implementation of BDD in an agile development team. This is due partly to the reasonable results from the performance tests that measured the speed of feedbacks as an efficiency test, partly to a high degree of both usability and reusability. This study is limited to few test scenarios due to shortage of time given for this course and therefore the performance results doesn’t express all kind of testing scenarios. / Kvalitetssäkring blir allt viktigare att genomföra i växande utvecklingsprojekt. Komponenterna som byggs måste testas regelmässigt för att garantera funktionaliteten av produkten som utvecklas. Leeroy tillämpar idag Scrum som utvecklingsmetod med manuella tester som genomförs när alla komponenter är färdigbyggda vid slutet av varje sprint. Detta leder till att testaren har kortare tid på sig för att utföra testscenarierna. Testaren utsätts därför för stress i sitt arbete då uppdraget kräver en hög noggrannhet med snabbhet för att effektivt försäkra att komponenterna hamnar i en produktionsmiljö. Syftet med undersökningen är att effektivisera nuvarande testprocesser genom att ersätta de manuella GUI-tester med automatiserade GUI- regressionstester som tillämpar BDD principer. Specificering av kraven genomfördes med hjälp av datainsamlingsmetoder som valdes till denna undersökning. Kravspecifikationerna användes dels för utvärdering av konstruktionen och dels för utvärdering av de erhållna resultaten. Resultaten visar att det är viktigt att både testaren och resten av det agila teamet samarbetar under varje sprint, då komponenterna utvecklas inkrementellt och testas regelmässigt. Resultaten visar vikten av att prioritera testerna som kommer att utvecklas för att säkra effektiviteten i arbetsflödet och även säkra att de viktigaste funktionerna testas först. Undersökningen har visat att både Cucumber och Puppeteer kan bidra till en smidig implementation av BDD i ett agilt utvecklingsteam. Detta beror dels på de rimliga resultaten från prestandatesterna som mäter effektiviteten, dels en hög användbarhetsgrad vad gäller både läsbarhet och skrivbarhet i syntaxen och dels en hög återanvändbarhetsgrad. Denna studie avgränsades till få antal testscenarier på grund av brist på tid i undersökningen och därför visar resultaten från prestandatesterna inte alla slags testscenarier.

This master thesis deals with application of an evolutionary algorithm in the creation of regression tests. In the first section, description of functional verification, verification methodology, regression tests and evolutionary algorithms is provided. In the following section, the evolutionary algorithm, the purpose of which is to achieve reduction of the number of test vectors obtained in the process of functional verification, is proposed. Afterwards, the proposed algorithm is implemented and a set of experiments is evaluated. The results are discussed.

Submitted by Geandra Rodrigues (geandrar@gmail.com) on 2018-07-20T12:18:30Z No. of bitstreams: 1 camilaacaciodepaiva.pdf: 9167548 bytes, checksum: 23bc00520b1453f40c0c4c739227c90b (MD5) / Approved for entry into archive by Adriana Oliveira (adriana.oliveira@ufjf.edu.br) on 2018-07-23T15:21:00Z (GMT) No. of bitstreams: 1 camilaacaciodepaiva.pdf: 9167548 bytes, checksum: 23bc00520b1453f40c0c4c739227c90b (MD5) / Made available in DSpace on 2018-07-23T15:21:00Z (GMT). No. of bitstreams: 1 camilaacaciodepaiva.pdf: 9167548 bytes, checksum: 23bc00520b1453f40c0c4c739227c90b (MD5) Previous issue date: 2018-06-27 / CAPES - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior / O software está cada vez mais presente no cotidiano das pessoas. Vários setores ou aspectos do ambiente são influenciados por ele. Desta forma, o desenvolvimento de software torna-se uma atividade crítica. Assim, o processo de teste se torna crucialmente importante, pois qualquer negligência pode refletir na qualidade do produto. Contudo, o cenário de desenvolvimento de software vem sofrendo mudanças a partir da necessidade de suprir demandas com maior agilidade e as exigências do mercado. É fundamental haver uma visão holística dos processos de desenvolvimento do software com o objetivo de gerar um ciclo de melhoria contínua. Tal visão é denominada como Engenharia de Software Contínua. A Engenharia de Software Contínua é caracterizada pelo uso do feedback de execuções para alcançar uma melhoria contínua e pela realização das atividades de maneira contínua. Feedback esse que pode ser fornecido através da proveniência de dados: descrição das origens de um dado e os processos pelos quais passou. Diante disso, este trabalho apresenta uma infraestrutura que tem como foco a captura e o armazenamento do histórico dos dados de execução do projeto, a previsão dos resultados dos testes de unidade através de algoritmos de previsão Logistic Regression, Naive Bayes e C4.5 Algorithm, e a disponibilização dos dados para aplicações externas. Além disso, oferece elementos de visualização que auxiliam na compreensão dos dados. Um experimento com dados de um projeto real foi realizado com o intuito de identificar a acurácia das previsões. / Software is increasingly present in people's daily lives. Various sectors and the environment are influenced by them. In this way, the development of software becomes a critical activity. Thus, the testing process becomes crucially important because any negligence can affect the quality of the product and the insecurity related to the use of the software. However, the software development scenario has undergone changes from the need to meet demands with greater agility and the demands of the market. It is fundamental to have a holistic view of the software development processes in order to generate a cycle of continuous improvement. Such a view is referred to as Continuous Software Engineering. Continuous Software Engineering is characterized by the use of feedback from executions to achieve continuous improvement and by performing activities on an ongoing basis. Feedback this can be provided through the provenance of data: description of the origins of a given and the processes by which it passed. This work presents an infrastructure that focuses on the capture and storage of the project execution data history, the prediction of the results of the unit tests through prediction algorithms: Logistic Regression, Naive Bayes and C4.5 Algorithm, and the provision of data for external applications. It also provides preview elements which help in understanding the data. An experiment with data from a real project was carried out in order to identify the accuracy of the prediction.

This thesis concerns automation of software and hardware testing using the robotic testing system called RQA. It aims to design an interface to produce test scenarios using the robotic testing system. Next it elaborates metrics measured during testing used for quality assurance of the system uder test. One of the topics covered is a visualization of test data and measurements for an easy overview of quality.

Made available in DSpace on 2014-12-17T15:48:10Z (GMT). No. of bitstreams: 1 JoaoMGCJ_DISSERT.pdf: 2157600 bytes, checksum: e625012e4cd3f5e995b205310bff8592 (MD5) Previous issue date: 2014-04-29 / Coordena??o de Aperfei?oamento de Pessoal de N?vel Superior / The main goal of Regression Test (RT) is to reuse the test suite of the latest version of a software in its current version, in order to maximize the value of the tests already developed and ensure that old features continue working after the new changes. Even with reuse, it is common that not all tests need to be executed again. Because of that, it is encouraged to use Regression Tests Selection (RTS) techniques, which aims to select from all tests, only those that reveal faults, this reduces costs and makes this an interesting practice for the testing teams. Several recent research works evaluate the quality of the selections performed by RTS techniques, identifying which one presents the best results, measured by metrics such as inclusion and precision. The RTS techniques should seek in the System Under Test (SUT) for tests that reveal faults. However, because this is a problem without a viable solution, they alternatively seek for tests that reveal changes, where faults may occur. Nevertheless, these changes may modify the execution flow of the algorithm itself, leading some tests no longer exercise the same stretch. In this context, this dissertation investigates whether changes performed in a SUT would affect the quality of the selection of tests performed by an RTS, if so, which features the changes present which cause errors, leading the RTS to include or exclude tests wrongly. For this purpose, a tool was developed using the Java language to automate the measurement of inclusion and precision averages achieved by a regression test selection technique for a particular feature of change. In order to validate this tool, an empirical study was conducted to evaluate the RTS technique Pythia, based on textual differencing, on a large web information system, analyzing the feature of types of tasks performed to evolve the SUT / O objetivo dos testes de regress?o (RT) ? reutilizar o conjunto de testes da ?ltima vers?o de um software em sua vers?o atual, para maximizar o valor dos testes j? desenvolvidos e garantir que antigas funcionalidades continuem corretas ap?s as novas modifica??es. Mesmo com o reuso, ? comum que nem todos os testes precisem ser executados novamente e para evitar o desnecess?rio, ? estimulada a utiliza??o de t?cnicas de sele??o dos testes de regress?o (RTS), que buscam selecionar dentre todos os testes, apenas aqueles capazes de revelar faltas, isto reduz custos e torna a pr?tica realmente atrativa para as equipes de teste. Diversos estudos recentes avaliam a qualidade da sele??o realizadas por t?cnicas de RTS, identificando qual delas apresenta melhores resultados atrav?s de m?tricas como a inclus?o e a precis?o. As t?cnicas de RTS deveriam buscar no sistema sob teste (SUT) por testes que revelem faltas, entretanto, como este ? um problema sem solu??o vi?vel, a alternativa ? buscar por testes que revelem as modifica??es, onde as faltas podem ocorrer. Contudo, tais modifica??es podem alterar o pr?prio fluxo de execu??o dos algoritmos, fazendo com que alguns testes n?o exercitem mais os mesmos trechos. Neste contexto, esta disserta??o de mestrado busca investigar se as modifica??es realizadas no SUT poderiam afetar a qualidade da sele??o dos testes realizada por uma RTS, e se sim, quais caracter?sticas apresentam as modifica??es que provocaram os erros, levando a RTS a incluir ou excluir testes erroneamente. Para tanto, foi desenvolvida uma ferramenta na linguagem Java para automatizar o c?lculo da inclus?o e precis?o m?dias alcan?adas por uma t?cnica de RTS para uma dada caracter?stica da modifica??o. A fim de validar a ferramenta, foi conduzido um estudo emp?rico para avaliar a t?cnica de RTS Pythia, baseada em diferencia??o textual, sobre um sistema de informa??o web de larga escala, analisando a caracter?stica dos tipos das tarefas realizadas para evoluir o SUT

Testning är en nödvändig men kostsam del av mjukvaruutveckling. Test utförs på olika abstraktionsnivåer och kan vara manuella eller automatiserade. På lägsta abstraktionsnivå, enhetsnivå, är automatiserad testning vanligt och relativt okomplicerat, medan systemtester är svårare att automatisera. I synnerhet gäller detta tester på ett grafiskt användargränssnitt (GUI) som kräver speciella verktyg.   Triona vill undersöka möjligheterna att automatisera regressionstester från GUI:t av sin produkt C-Load, en molnbaserad webbtjänst för avtalsbaserad transportbokning.    Det primära syftet med denna fallstudie är att med en anpassad urvalsprocess utvärdera ett möjligt verktyg i förhållande till C-Load-förvaltningens förväntningar på automatiserad GUI testning (AGT) och att utifrån resultatet föreslå hur C-Loadförvaltningen kan gå vidare med val av verktyg för AGT. För att uppfylla syftet användes litteraturstudier, intervjuer och observationer av praktiska test.    Verktyg för GUI-testning kan delas in i tre huvudkategorier: skriptbaserade, modellbaserade och skriptlösa. Baserat på tidigare forskning drogs slutsatsen att ett skriptbaserat verktyg där koden i testskripten skrivs manuell bäst passar C-Loadförvaltningens krav och förutsättningar. Det mest använda verktyget av denna typ, Selenium WebDriver, utvärderades kvalitativt gentemot identifierade krav.    Av tidigare forskning framgår att vanliga utmaningar med skriptbaserade GUI-tester är att arbetsinsatsen för att skapa och underhålla testskript är stor och att testen kan vara opålitliga. Dessa problem framkom också i studiens intervjuer och observationer.   Slutsatsen är att det vore möjligt att automatisera regressionstester av C-Load med hjälp av Selenium Webdriver, och att det på sikt skulle kunna frigöra tid. Initialt krävs dock en omfattande insats för att implementera automatiserade tester i förvaltningen och Selenium Webdriver uppfyller bara delvis C-Load-förvaltningens förväntningar på AGT. C-Load-förvaltningen rekommenderas att utvärdera fler verktyg innan beslut fattas. I en kommande urvalsprocess bör Triona beakta hur väl olika verktyg fungerar i förhållande till moderna webbramverk. / Testing is a necessary but costly part of software development. Tests are performed at different abstraction levels and can be either manual or automated. On the lowest level of abstraction, where unit testing is performed, automated testing is commonplace and relatively uncomplicated, whereas system testing is more difficult to automate. This is especially true for GUI-testing, which requires special tools.      Triona wished to investigate possibilities to automate regression testing of the GUI for its C-load product, which is a Cloud-based web-service for contract-based transport booking.      The purpose of this case study was to evaluate one tool for automated GUI-testing (AGT) against the C-Load team’s expectations on AGT, and based on the result recommend Triona how to proceed in the process of implementing AGT. Literature studies, observations and interviews were conducted to fulfil the purpose.      GUI-testing tools can be classified into three categories: script-based, model-based and scriptless. One conclusion was that a script-based tool, where test scripts are manually coded would best suit Triona’s needs. The most used tool in that category, Selenium WebDriver, was tested and evaluated against requirements.       Prior research shows that common challenges encountered when using script-based GUItests are the workload required to create and maintain test scripts and that the tests can deliver inconsistent or “flaky” results. These challenges were confirmed during our analysis.       Our conclusion is that it is possible to automate C-Load regression tests with Selenium WebDriver, and that it would eventually free up time. However, a considerable effort is initially required to implement automated testing. Selenium Webdriver only partly fulfills the C-Load team’s expectations on AGT. Before a decision is taken, the C-Load team should evaluate more tools. When evaluating tools for AGT, Triona should take note that Selenium Webdriver can be deficient when it comes to testing applications based on modern web frameworks.