Tese de Mestrado Integrado. Engenharia Informática e Computação. Faculdade de Engenharia. Universidade do Porto. 2012

When we survey hardware design groups, we can find that it is now dedicated to verification between 60 to 80 percent. According to the instruction set architecture information should be a feasible and reasonable way for generating the test pattern to verify the function of a microprocessor. In this these, we¡¦ll present an instruction test pattern (for microprocessors) generation method based on the instruction set architecture. It can help the users to generate the instruction test pattern efficiently. The generation flow in this thesis contains three major flows: individual instruction, instruction pair, and manual generation. They are used for different verification cases. The ¡§individual instruction¡¨ could be used for verifying the functions of each implemented instructions. The ¡§instruction pair¡¨ could be used for verifying the interaction of instruction execution in a pipeline for a HDL implementation of a microprocessor. The ¡§manual generation¡¨ could be used to verify some corner cases (behaviors) of the microprocessor. As the quality of our test pattern, we generate some patterns for 32-bits instruction (ARM instruction sets and SPARC instruction sets) and use them to verify a synthesizable RTL core. With some handwriting test pattern (34.7%), our automatic generation method can approach 100% HDL code coverage of the microprocessor design. We use the HDL code coverage as the reference of test pattern quality. Because our generation method is based on the instruction field, we can describe most instruction set for the generator. Hence, our generation method can retarget to most instruction set architecture without modifying the generator. Besides the RISC instructions, even the CISC instructions could be generated.

No description available.

How can you tell whether your test suites are reliable? This is often done through the use of coverage criterion that would define a set of requirements that the test suites need to fulfill in order to be considered reliable. The most widely used criterion is those referred to as code coverage, where the degree to which the code base is covered is used as an arbitrary measure of how good the test suites are. Achieving high coverage would indicate an adequate test suite i.e. reliable according to the standards of code coverage. However, covering a line of code does not necessarily mean that it has been tested. Thus, code coverage can only tell you what parts of the code base have not been tested, opposed to what have been tested. Mutation testing on the other hand is an approach to evaluate the adequacy of test suites through their fault detection ability, rather than how much of the code base they cover.   This thesis performs mutation analysis on a project with inadequate code coverage. The present testing effort on unit level is evaluated and the cost and benefits of adopting mutation testing as a testing method is evaluated. / Hur vet man när tester är tillförlitliga? Ofta använder man sig av täckningskriterium som definierar en uppsättning krav som tester måste uppfylla för att betraktas som pålitlig. Det mest använda kriterier är de som kallas kodtäckning, där graden till vilken kodbasen är täckt används som ett mått av pålitlighet av tester. Hög täckning indikerar adekvat tester, dvs pålitlig enligt kodtäckning. Men täckning av en kodlinje betyder inte nödvändigtvis att den har testats. Koddekning kan således bara visa vilka delar av kodbasen som inte har testats, snarare än vad som har testats. Mutation testing å andra hand är ett sätt att utvärdera testers effektivitet genom deras felsökningsförmåga, snarare än hur mycket av kodbasen de täcker.   Denna examensarbete utför mutationsanalys på ett projekt med otillräcklig koddekning. Kvalite av nuvarande tester på enhetsnivå utvärderas och kostnaden och fördelar för att anta mutation testning som en testmetod utforskas.

Depuração é a atividade responsável pela localização e correção de defeitos gerados durante o desenvolvimento de programas. A depuração ocorre devido à atividade de teste bem-sucedida, na qual falhas no comportamento do programa são reveladas, indicando a existência de defeitos. Diversas técnicas têm sido propostas para automatizar a tarefa de depuração de programas. Algumas delas utilizam heurísticas baseadas em informações de cobertura obtidas da execução de testes. O objetivo é indicar trechos de código do programa mais suspeitos de conter defeitos. As informações de cobertura mais usadas em depuração automatizada são baseadas no teste estrutural de unidade. A cobertura de integração, obtida por meio da comunicação entre as unidades de um programa, pode trazer novas informações sobre o código executado, possibilitando a criação de novas estratégias para a tarefa de localização de defeitos. Este trabalho apresenta uma nova técnica de localização de defeitos chamada Depuração de programas baseada em Cobertura de Integração (DCI). São apresentadas duas coberturas de integração baseadas nas chamadas de métodos de um programa. Essas coberturas são usadas para a proposição de roteiros de busca dos defeitos a partir dos métodos considerados mais suspeitos. As informações de cobertura de unidade são então utilizadas para a localização dos defeitos dentro dos métodos. A DCI também utiliza uma nova heurística para atribuição de valores de suspeição a entidades de integração estática dos programas como pacotes, classes e métodos, fornecendo também um roteiro para a procura dos defeitos. Os experimentos realizados em programas reais mostram que a DCI permite realizar a localização de defeitos de forma mais eficaz do que o uso de informações de cobertura de unidade isoladamente. / Debugging is the activity responsible for localizing and fixing faults generated during software development. Debugging occurs due to a successful testing activity, in which failures in the behavior of the program are revealed, indicating the existence of faults. Several techniques have been proposed to automate the debugging tasks, especially the fault localization task. Some techniques use heuristics based on coverage data obtained from the execution of tests. The goal is to indicate program code excerpts more likely to contain faults. The coverage data mostly used in automated debugging is based on white-box unit testing. Integration coverage data, obtained from the communication between the units of a program, can bring about new information with respect to the executed code, which allows new strategies to the fault localization task to be devised. This work presents a new fault localization technique called Debugging based on Integration Coverage (DIC). Two integration coverages based on method invocations are presented. These coverages are used to propose two search strategies that provides a roadmap to locate faults by investigating the more suspicious methods. The unit coverage information are used to search the faulty statement inside the suspicious methods. The DIC technique also proposes a heuristic that assigns suspiciousness values to static integration entities of the programs, namely, packages, classes, and methods. This heuristic also provides a roadmap to search for the faults. Experiments using real programs show that DIC is more effective to locate faults than solely using unit coverage information.

Depuração é a atividade responsável pela localização e correção de defeitos gerados durante o desenvolvimento de programas. A depuração ocorre devido à atividade de teste bem-sucedida, na qual falhas no comportamento do programa são reveladas, indicando a existência de defeitos. Diversas técnicas têm sido propostas para automatizar a tarefa de depuração de programas. Algumas delas utilizam heurísticas baseadas em informações de cobertura obtidas da execução de testes. O objetivo é indicar trechos de código do programa mais suspeitos de conter defeitos. As informações de cobertura mais usadas em depuração automatizada são baseadas no teste estrutural de unidade. A cobertura de integração, obtida por meio da comunicação entre as unidades de um programa, pode trazer novas informações sobre o código executado, possibilitando a criação de novas estratégias para a tarefa de localização de defeitos. Este trabalho apresenta uma nova técnica de localização de defeitos chamada Depuração de programas baseada em Cobertura de Integração (DCI). São apresentadas duas coberturas de integração baseadas nas chamadas de métodos de um programa. Essas coberturas são usadas para a proposição de roteiros de busca dos defeitos a partir dos métodos considerados mais suspeitos. As informações de cobertura de unidade são então utilizadas para a localização dos defeitos dentro dos métodos. A DCI também utiliza uma nova heurística para atribuição de valores de suspeição a entidades de integração estática dos programas como pacotes, classes e métodos, fornecendo também um roteiro para a procura dos defeitos. Os experimentos realizados em programas reais mostram que a DCI permite realizar a localização de defeitos de forma mais eficaz do que o uso de informações de cobertura de unidade isoladamente. / Debugging is the activity responsible for localizing and fixing faults generated during software development. Debugging occurs due to a successful testing activity, in which failures in the behavior of the program are revealed, indicating the existence of faults. Several techniques have been proposed to automate the debugging tasks, especially the fault localization task. Some techniques use heuristics based on coverage data obtained from the execution of tests. The goal is to indicate program code excerpts more likely to contain faults. The coverage data mostly used in automated debugging is based on white-box unit testing. Integration coverage data, obtained from the communication between the units of a program, can bring about new information with respect to the executed code, which allows new strategies to the fault localization task to be devised. This work presents a new fault localization technique called Debugging based on Integration Coverage (DIC). Two integration coverages based on method invocations are presented. These coverages are used to propose two search strategies that provides a roadmap to locate faults by investigating the more suspicious methods. The unit coverage information are used to search the faulty statement inside the suspicious methods. The DIC technique also proposes a heuristic that assigns suspiciousness values to static integration entities of the programs, namely, packages, classes, and methods. This heuristic also provides a roadmap to search for the faults. Experiments using real programs show that DIC is more effective to locate faults than solely using unit coverage information.

Telecom base stations are a critical part of society's information infrastructure. To ensure high quality base station software, automated testing is an important part of development. Ericsson measures the quality of automated tests with statement coverage, counting the number of statements executed by a test suite. Alone, however, statement coverage does not guarantee test quality. Mutation testing is a technique to improve test quality by injecting faults and verifying that test suites detect them. This thesis investigates whether mutation testing is a viable way to increase the reliability of test suites for base station software at Ericsson. Using the open-source mutation testing tool MiLu, we describe a practical method of using mutation testing that is viable for daily development. We also describe how mutation testing reveals a numbers of potential errors in the production code that current test suites miss even though they have very good statement coverage.

Regression testing is performed after existing source code has been modified to verify that no new faults have been introduced by the changes. Test case selection can be used to reduce the effort of regression testing by selecting a smaller subset of the test suite for later execution. Several criteria and objectives can be used as constraints that should be satisfied by the selection process. One common criteria is function coverage, which can be represented by a coverage matrix that maps test cases to methods under test. The process of generating and evaluating these matrices can be very time consuming for large matrices since their complexity increases exponentially with the number of tests included. To the best of our knowledge, no techniques for reducing execution matrix size have been proposed. This thesis develops a matrix-reduction technique based on analysis of call stack data. It studies the effects of limiting the call stack depth in terms of coverage accuracy, matrix size, and generation costs. Further, it uses a tool that can instrument Java projects using Java’s instrumentation API to collect coverage information on open-source Java projects for varying depth limits of the call stack. Our results show that the stack depth limit can be significantly reduced while retaining high coverage and that matrix size can be decreased by up to 50%. The metric we used to indicate the difficulty of splitting up the matrix closely resembled the curve for coverage. However, we did not see any significant differences in execution time for lower depth limits.

No description available.

Viljan att hålla en hög kvalitet på den kod som skrivs vid utveckling av system och applikationerär inte något nytt i utvecklingsvärlden. Flera större företag använder sig av olika mått för attmäta kvaliteten på koden i sina system med målet att hålla en hög driftsäkerhet.Trafikverket är en statlig myndighet som ansvarar för driften av bland annat de system somhåller igång Sveriges järnvägsnät. Eftersom systemen fyller en viktig del i att säkra driften ochse till att tågpositioner, planering av avgångar och hantering av driftstörningar fungerar dygnetrunt för hela landet anser de att det är viktigt att sträva efter att hålla en hög kvalitet påsystemen.Syftet med det här examensarbetet var att ta reda på vilka mått som kan vara möjliga attanvända under systemutvecklingsprocessen för att mäta kvaliteten på kod och hur måtten kananvändas för att öka kvaliteten på IT-lösningar. Detta för att redan på ett tidigt stadie kunnamäta kvaliteten på den kod som skrivs i både befintliga och nyutvecklade system.Studien är en fallstudie som utfördes på Trafikverket, de olika måtten som undersöktes varcode coverage, nivån på maintainability index och antalet inrapporterade incidenter för varjesystem. Mätningar utfördes på sju av Trafikverkets system som i analysen jämfördes motantalet rapporterade incidenter. Intervjuer utfördes för att ge en bild över hur arbetssättet vidutveckling kan påverka kvaliteten. Genom litteraturstudier kom det fram ett mått som inte kundeanvändas praktiskt i det här fallet men är högst intressant, detta är cyclomatic complexity somfinns som en del av maintainability index men som även separat påverkar möjligheten att skrivaenhetstest.Resultaten av studien visar att måtten är användbara för ändamålet men bör inte användassom enskilda mått för att mäta kvalitet eftersom de fyller olika funktioner. Det är viktigt attarbetssättet runt utveckling genomförs enligt en tydlig struktur och att utvecklarna både harkunskap om hur man arbetar med enhetstest och följer kodprinciper för strukturen. Tydligakopplingar mellan nivån på code coverage och inflödet av incidenter kunde ses i de undersöktasystemen där hög code coverage ger ett lägre inflöde av incidenter. Ingen korrelation mellanmaintainability index och incidenter kunde hittas. / The desire to ascertain a high level of quality on the code written during the development of systems and applications is not something new in the system development world. Several larger companies use different kinds of metrics to measure the quality of the code in their systems with the goal of maintaining high reliability and quality.Trafikverket is a government authority responsible for the operation of the system that keeps the Swedish railroad running. Their systems play an important part in ensuring the operation and to ensure that train positions, the planning of departures and error handling works around the clock for the entire country, they find it important and strive to maintain the high quality of the systems.The aim of this thesis was to find out which measurements may be possible to use during the system development process to measure the quality of the code and how measurements can be used to increase the quality of IT solutions. It should be possible to measure the quality of the code that is written in both existing and newly developed systems at an early stage. The study is a case study conducted at Trafikverket, the metrics that were examined were code coverage, the level of maintainability index and the number of reported incidents for each system. Measurements were performed on seven of Trafikverket's systems. In the analysis the measurements were compared to the number of reported incidents. Interviews were conducted to provide a picture of how the operation of the development may affect the quality. Through literature studies we discovered a metric that could not be used practically in this case, this was cyclomatic complexity, it is available as part of the maintainability index, but also separately affects the ability to write unit tests. The results of the study show that the measurements are useful for this purpose but should not be used as individual metrics to measure quality because they all have their own function. A clear link between the level of the code coverage and the number of incidents could be observed in the investigated systems where high code coverage provides a lower rate of incidents. No correlation between maintainability index and incidents could be found.