Software testing verifies the program's functional behavior, one important process when engineering critical software. Measuring the degree of testing is done with code coverage, describing the amount of production code affected by tests. Both concepts are extensively used for industrial systems. Previous research has shown that gathering and analyzing test coverages becomes problematic on large-scale systems. Here, development experience, implementation feasibility, coverage measurements and analysis method are explored; providing potential solutions and insights into these issues. Outlined are methods for constructing and integrating such gathering and analysis system in a large-scale project, along with the problems encountered and given remedies. Instrumentations for gathering coverage information affect performance negatively, these measurements are provided. Since large-scale test suite measurements are quite lacking, the line, branch, and function criteria are presented here. Finally, an analysis method is proposed, by using coverage set operations and Jaccard indices, to find test similarities. Results gathered imply execution time was significantly affected when gathering coverage, [2.656, 2.911] hours for instrumented software, originally between [2.075, 2.260] on the system under test, given under the alpha = 5% and n = 4, while both processor & memory usages were inconclusive. Measured criteria were (59.3, 70.7, 24.6)% for these suites. Analysis method shows potential areas of test redundancy.

Actuellement, le développement d’un logiciel de taille industriel repose généralement surdes tests ou des preuves unitaires pour garantir rigoureusement ses exigences. En outre, il adéjà été montré que l’utilisation combinée du test et de la preuve unitaires est plus efficaceque l’utilisation d’une seule de ces deux techniques. Néanmoins, un ingénieur en vérificationhésite encore à utiliser ces deux techniques conjointement, faute d’une notion de couverturecommune au test et à la preuve. Définir une telle notion est l’objet de cette thèse.En effet, nous introduisons une nouvelle couverture, appelée « couverture label-mutant ».Elle permet de représenter les critères de couverture structurelle habituels du test, comme lacouverture des instructions, la couverture des branches ou la couverture MC/DC et de décidersi le critère choisi est satisfait en utilisant une technique de vérification formelle, qu’elle soitpar test, par preuve ou par une combinaison des deux. Elle permet également de représenterles critères de couverture fonctionnelle. Nous introduisons aussi dans cette thèse une méthodereposant sur des outils automatiques de test et de preuve pour réduire l’effort de vérificationtout en satisfaisant le critère de couverture choisi. Cette méthode est mise en oeuvre au seinde la plateforme d’analyse de code C (Frama-C), fournissant ainsi à un ingénieur un moyenopérationnel pour contrôler et réaliser la vérification qu’il souhaite. / Currently, industrial-strength software development usually relies on unit testing or unitproof in order to ensure high-level requirements. Combining these techniques has already beendemonstrated more effective than using one of them alone. The verification engineer is yetnot been to combine these techniques because of the lack of a common notion of coverage fortesting and proving. Defining such a notion is the main objective of this thesis.We introduce here a new notion of coverage, named « label-mutant coverage ». It subsumesmost existing structural coverage criteria for unit testing, including statement coverage,branch coverage or MC/DC coverage, while allowing to decide whether the chosen criterionis satisfied by relying on a formal verification technique, either testing or proving or both.It also subsumes functional coverage criteria. Furthermore, we also introduce a method thatmakes use of automatic tools for testing or proving in order to reduce the verification costwhile satisfying the chosen coverage criterion. This method is implemented inside Frama-C, aframework for verification of C code (Frama-C). This way, it offers to the engineer a way tocontrol and to perform the expected verifications.

Test coverage is an important parameter of analyzing how well the product is being tested in any domain within the IT industry. Unit testing is one of the important processes that have gained even more popularity with the rise in Test driven development (TDD) culture.This degree project, conducted at NASDAQ Technology AB, analyzes the existing unit tests in one of the products, and compares various coverage models in terms of quality. Further, the study examines the factors that affect code coverage, presents the best practices for unit testing, and a proven test process used in a real world project.To conclude, recommendations are given to NASDAQ based on the findings of this study and industry standards. / Testtäckning är en viktig parameter för att analysera hur väl en produkt är testad inom alla domäner i IT-industrin. Enhetstestning är en av de viktiga processerna som har ökat sin popularitet med testdriven utveckling. Detta examensarbete, utfört på NASDAQ Technology AB, analyserar de befintliga testen i en av produkterna, och jämför olika kvalitetsmodeller. Vidare undersöker undersökningen de faktorer som påverkar koddekning, presenterar de bästa metoderna för enhetstestning och en beprövad testprocess som används i ett verkligt världsprojekt. Avslutningsvis ges rekommendationer till NASDAQ baserat på resultaten från denna studie och industristandarder.

A number of logical code coverage criteria have been used throughout the years in the testing of safety-critical software. Kaminski, et al. proposed Relational Operator Replacement Global (RORG), a method to bring benefits from ROR mutation to Modified Condition / Decision Coverage (MCDC), which is widely used in the avionics industry. However, there is a lack of studies in the industry to support this method. In this thesis, we report on the results of applying RORG to avionic code, augmenting an MCDC adequate test set to satisfy RORG, evaluating its ability to find real faults in industrial software. Conclusions drawn from this thesis are: (1) Faults in relational operators in avionic code are rare, no faults were found in this study. (2) 24% of the relational operators in our study would require additional software requirements to be verified for RORG coverage. (3) 37% of the relational operators in our study were infeasible to test due to program semantics. (4) 84% of the tests added covered enumeration comparisons.

Submitted by JÚLIO HEBER SILVA (julioheber@yahoo.com.br) on 2016-12-05T15:46:39Z No. of bitstreams: 2 Dissertação - Daniel Gomes de Oliveira - 2016.pdf: 1447085 bytes, checksum: f382ec268ae42480adeee8f03e5ccda2 (MD5) license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) / Approved for entry into archive by Jaqueline Silva (jtas29@gmail.com) on 2016-12-13T15:32:34Z (GMT) No. of bitstreams: 2 Dissertação - Daniel Gomes de Oliveira - 2016.pdf: 1447085 bytes, checksum: f382ec268ae42480adeee8f03e5ccda2 (MD5) license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) / Made available in DSpace on 2016-12-13T15:32:34Z (GMT). No. of bitstreams: 2 Dissertação - Daniel Gomes de Oliveira - 2016.pdf: 1447085 bytes, checksum: f382ec268ae42480adeee8f03e5ccda2 (MD5) license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) Previous issue date: 2016-09-29 / Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES / Considering the high cost and large amount of time demanded by the activity generation tests in the software development process, the need a proposal to reduce both the time spent as the related costs testing activities is necessary. In this context, the use of tools or processes that make the activities of generation of more agile testing, less costly and meet demands for precision are key to companies operating in software development market can achieve their goals. Based on these information comes to questions regarding how to go about adopting a process that makes possible the achievement of objectives in order to meet the results mentioned previously, even with the difficulties of generating test data as a result of of programs input areas are infinite. There are different tools that use various strategies for generating test data, however, lacks evidence as the quality of these tools. In this context, the aim of this work is conducting an experimental evaluation of some automatic test data generators to identify which one offers the best cost / benefit in terms of effective in detecting defects number of generated test data, code coverage demanded by test data, and generation time of testing. At second step a third tool was included along manually generated tests. New test sets using three automatic generators and included the manually -generated sets project were generated. Finally, results were presented in terms of effectiveness and efficiency through the comparison between the four test sets . / Considerando o alto custo e a grande quantidade de tempo demandada pela atividade de criação de casos de testes dentro do processo de desenvolvimento de software. A utilização de ferramentas ou procedimentos que tornem o processo de geração de dados de testes mais ágil, menos oneroso e que atendam demandas por precisão se tornam fundamentais para que as empresas atuantes no mercado de desenvolvimento de software possam atingir seus objetivos. Com base nessas informações, surge a dúvida relacionada a como proceder para adotar um processo de desenvolvimento e teste de software que tornem possíveis o alcance dos objetivos de forma a atender os resultados mencionados anteriormente, mesmo com as dificuldades de gerar dados de teste em decorrência dos domínios de entrada dos programas serem em geral infinitos. O objetivo do presente trabalho é conduzir uma avaliação experimental de geradores automáticos de dados de teste visando identificar qual deles apresenta a melhor relação custo/benefício em termos de eficácia em detectar defeitos, número de dados de teste gerados e cobertura de código determinada pelos conjuntos de teste. A pesquisa foi dirigida em duas etapas: na primeira, dois geradores foram avaliados em relação a um conjunto de 32 programas Java e os resultados obtidos indicam que, de maneira geral, o gerador CodePro foi o que apresentou a melhor relação custo benefício frente ao Randoop; na segunda, foi inclusa uma terceira ferramenta, juntamente a testes gerados de forma manual. Foram gerados novos conjuntos de teste utilizando os três geradores automáticos e incluso ao projeto conjuntos gerados de forma manual. Ao final, foram apresentados os resultados em termos de eficácia e eficiência por meio dos comparativos entre os quatro conjuntos de teste.