Submitted by Lucienne Costa (lucienneferreira@ufcg.edu.br) on 2018-08-30T18:41:36Z No. of bitstreams: 1 INDY PAULA SOARES CORDEIRO E SILVA – DISSERTAÇÃO (PPGCC) 2018.pdf: 1399246 bytes, checksum: ca2d8bfa9c9492d9d2d97a1485afad82 (MD5) / Made available in DSpace on 2018-08-30T18:41:36Z (GMT). No. of bitstreams: 1 INDY PAULA SOARES CORDEIRO E SILVA – DISSERTAÇÃO (PPGCC) 2018.pdf: 1399246 bytes, checksum: ca2d8bfa9c9492d9d2d97a1485afad82 (MD5) Previous issue date: 2018-02-22 / Capes / Refatoramentos normalmente exigem testes de regress˜ao para verificar se as mudanc¸as aplicadas ao c´odigo, preservaram o comportamento original do mesmo. Geralmente, ´e dif´ıcil definir um conjunto de testes que seja efetivo para esta tarefa, uma vez que o refatoramento n˜ao ´e frequentemente aplicado em etapas isoladas. Al´em disso, as edic¸ ˜oes de refatoramento podem ser combinadas com outras edic¸ ˜oes no c´odigo. Nesse sentido, a gerac¸ ˜ao de casos de teste pode contribuir para essa tarefa, analisando sistematicamente o c´odigo e fornecendo uma ampla gama de casos de teste que abordam diferentes construc¸ ˜oes. No entanto, uma s´erie de estudos apresentados na literatura mostram que as ferramentas atuais ainda n˜ao s˜ao eficazes com relac¸ ˜ao `a detecc¸ ˜ao de faltas, particularmente faltas de refatoramento. Com base nisto, apresentamos dois estudos emp´ıricos que aplicaram as ferramentas Randoop e Evosuite para gerar suites de testes de regress˜ao, com foco na edic¸ ˜ao de refatoramento do tipo extract method. Com base nos resultados dos estudos, identificamos fatores que podem influenciar o desempenho das ferramentas para efetivamente testar a edic¸ ˜ao. Para validar nossos achados, apresentamos um conjunto de modelos de regress˜ao que associam a presenc¸a desses fatores `a capacidade do conjunto de testes, de detectar faltas relacionadas `a edic¸ ˜ao de refatoramento. E por fim, apresentamos a REFANALYZER, que ´e uma ferramenta que objetiva ajudar os desenvolvedores a decidir quando confiar em suites geradas automaticamente, para validar refatoramento do tipo extract method. / Refactoring typically require regression testers to verify that the changes applied to the code have preserved the original behavior of the code. Generally, it is difficult to define a set of tests that is effective for this task, since refactoring is not the weight applied in isolated steps. Also, since refactoring issues can be combined with other issues, without code. In this sense, a generation of test cases can contribute to this task by systematically analyzing the code and defining a wide range of test cases that address different constructs. However, a number of studies are not available but are not very effective in detecting faults, particularly refactorings. Based on this, we present two empirical studies that have applied as Randoop and Evosuite tools to generate regression test suites, focusing on the edition of refactoring of the extract method type. Based on the results of the studies, we identified factors that can influence the performance of the tools to effectively test the edition. To validate our findings, we present a set of control models that associate a solution with the ability of the set of tests, of spoken faults related to the refactoring edition. And finally, we present a REFANALYZER, which is a tool that is a solution to what needs to be solved when it is automatically managed, to validate refactoring of the extract method type.

A utilização desta abordagem no ciclo de desenvolvimento de uma aplicação web traz algumas vantagens, como ser necessário gerar manualmente apenas o modelo de comportamento de cada funcionalidade da aplicação, (pois os demais artefatos são gerados automaticamente), consumindo menos tempo e estando menos sujeitos a erros, além de prevenir diferentes interpretações dos requisitos pelos stakeholders, desenvolvedores e testadores. O tempo despendido na especificação dos modelos é compensado pelo tempo economizado com a geração dos cenários e do código de testes. / The testing activity throughout software development is fundamental to the pursuit of software quality and reliability, finding faults to be removed. However, despite its importance, software testing is often an underutilized phase in software development. Moreover, tests are proved to be expensive, difficult and problematic when not done in the appropriate way. A new paradigm for software testing is model-driven testing (MDT), which can be defined as software testing where test cases are derived from a model that describes some aspects of the system being tested, such as behavior, for example. This description, often using UML diagrams and/or its profiles, can be processed to produce a set of test cases. Software specifications based on usage scenarios expressed by appropriate UML diagrams are considered significant and effective, because they describe the system’s requirements from an intuitive and visual perspective. Thus, they can be used for the description of acceptance tests, which validate that the system meets user requirements. These specifications also facilitate the automation of this kind of test. Test automation can decrease time spent on testing, thereby reducing the cost of this activity. Thus, this work proposes an approach for automated generation of acceptance tests from U2TP (the UML 2.0 test profile) diagrams for web applications, based on behavior driven development (BDD) paradigm, obtaining acceptance scenarios and executable test code supported by an acceptance testing automation framework. This approach was applied on an actual development environment, by means of an experiment. Using this approach in an web application development cycle has some advantages, such as being required only to manually generate the model of behavior of each application functionality (because other artifacts are generated automatically), thus being less time consuming and less prone to errors, and preventing different interpretations of requirements by stakeholders, developers and testers. The time spent at the models’ specification is compensated by the time saved with the generation of scenarios and test code.

A utilização desta abordagem no ciclo de desenvolvimento de uma aplicação web traz algumas vantagens, como ser necessário gerar manualmente apenas o modelo de comportamento de cada funcionalidade da aplicação, (pois os demais artefatos são gerados automaticamente), consumindo menos tempo e estando menos sujeitos a erros, além de prevenir diferentes interpretações dos requisitos pelos stakeholders, desenvolvedores e testadores. O tempo despendido na especificação dos modelos é compensado pelo tempo economizado com a geração dos cenários e do código de testes. / The testing activity throughout software development is fundamental to the pursuit of software quality and reliability, finding faults to be removed. However, despite its importance, software testing is often an underutilized phase in software development. Moreover, tests are proved to be expensive, difficult and problematic when not done in the appropriate way. A new paradigm for software testing is model-driven testing (MDT), which can be defined as software testing where test cases are derived from a model that describes some aspects of the system being tested, such as behavior, for example. This description, often using UML diagrams and/or its profiles, can be processed to produce a set of test cases. Software specifications based on usage scenarios expressed by appropriate UML diagrams are considered significant and effective, because they describe the system’s requirements from an intuitive and visual perspective. Thus, they can be used for the description of acceptance tests, which validate that the system meets user requirements. These specifications also facilitate the automation of this kind of test. Test automation can decrease time spent on testing, thereby reducing the cost of this activity. Thus, this work proposes an approach for automated generation of acceptance tests from U2TP (the UML 2.0 test profile) diagrams for web applications, based on behavior driven development (BDD) paradigm, obtaining acceptance scenarios and executable test code supported by an acceptance testing automation framework. This approach was applied on an actual development environment, by means of an experiment. Using this approach in an web application development cycle has some advantages, such as being required only to manually generate the model of behavior of each application functionality (because other artifacts are generated automatically), thus being less time consuming and less prone to errors, and preventing different interpretations of requirements by stakeholders, developers and testers. The time spent at the models’ specification is compensated by the time saved with the generation of scenarios and test code.

A utilização desta abordagem no ciclo de desenvolvimento de uma aplicação web traz algumas vantagens, como ser necessário gerar manualmente apenas o modelo de comportamento de cada funcionalidade da aplicação, (pois os demais artefatos são gerados automaticamente), consumindo menos tempo e estando menos sujeitos a erros, além de prevenir diferentes interpretações dos requisitos pelos stakeholders, desenvolvedores e testadores. O tempo despendido na especificação dos modelos é compensado pelo tempo economizado com a geração dos cenários e do código de testes. / The testing activity throughout software development is fundamental to the pursuit of software quality and reliability, finding faults to be removed. However, despite its importance, software testing is often an underutilized phase in software development. Moreover, tests are proved to be expensive, difficult and problematic when not done in the appropriate way. A new paradigm for software testing is model-driven testing (MDT), which can be defined as software testing where test cases are derived from a model that describes some aspects of the system being tested, such as behavior, for example. This description, often using UML diagrams and/or its profiles, can be processed to produce a set of test cases. Software specifications based on usage scenarios expressed by appropriate UML diagrams are considered significant and effective, because they describe the system’s requirements from an intuitive and visual perspective. Thus, they can be used for the description of acceptance tests, which validate that the system meets user requirements. These specifications also facilitate the automation of this kind of test. Test automation can decrease time spent on testing, thereby reducing the cost of this activity. Thus, this work proposes an approach for automated generation of acceptance tests from U2TP (the UML 2.0 test profile) diagrams for web applications, based on behavior driven development (BDD) paradigm, obtaining acceptance scenarios and executable test code supported by an acceptance testing automation framework. This approach was applied on an actual development environment, by means of an experiment. Using this approach in an web application development cycle has some advantages, such as being required only to manually generate the model of behavior of each application functionality (because other artifacts are generated automatically), thus being less time consuming and less prone to errors, and preventing different interpretations of requirements by stakeholders, developers and testers. The time spent at the models’ specification is compensated by the time saved with the generation of scenarios and test code.

Software testing is an important process for ensuring the quality of the software. As the complexity of the software increases, traditional means of manual testing becomes increasingly more complex and time consuming. In most embedded systems, designing software with as few errors as possible is often critical. Resource usage is also of concern for proper behavior because of the very nature of embedded systems.  To design reliable and energy-efficient systems, methods are needed to detect hot points of consumption and correct them prior to deployment. To reduce testing effort, Model-based testing can be used which is one testing method that allows for automatic testing of model based systems. Model-based testing has not been investigated extensively for revealing resource usage anomalies in embedded systems. UPPAAL SMC is a statistical model checking tool which can be used to model the system’s resource usage. Currently UPPAAL SMC lacks the support for performing automatic test generation and test selection. In this thesis we provide this support with a framework for automatic test generation and test selection using mutation analysis, a method for minimizing the generated test suite while maximizing the fault coverage and a tool implementing the framework on top of the UPPAAL SMC tool. The thesis also evaluates the framework on a Brake by Wire industrial system. Our results show that we could for a Brake-by-wire system, simulated on a consumer processor with five mutants, in best case find a test case that achieved 100% mutation score within one minute and confidently identify at least one test case that achieved full mutation score within five minutes. The evaluation shows that this framework is applicable and relatively efficient on an industrial system for reducing continues resource usage target testing effort.

Les travaux de recherche menés au cours de cette thèse s'inscrivent dans le cadre de la modélisation, de la vérification et de la validation de systèmes complexes, critiques et multi-physiques. Ces travaux visent à combler l'écart d'abstraction entre les modèles haut-niveau, point de départ des processus MBSE (Model-Based Systems Engineering), et la simulation temps réel, clef de voûte des approches In-the-Loop. Dans ce contexte, nous proposons d'unifier, au sein d'un même modèle SysML, les aspects continus d'un système, permettant de générer de manière automatique un modèle Modelica de plus bas niveau directement exécutable (simulation), et les aspects discrets, permettant l'animation et la génération de tests par des solveurs de contraintes. Les travaux réalisés au cours de cette thèse ont permis l'étude et la réalisation d'une chaîne outillée originale permettant de simuler et de tester ce type de systèmes à partir de modèles SysML en contexte In-the-Loop. Cette démarche a été validée par deux cas d'étude concrets issus de la recherche. Le premier, issu du projet ANR Smart Blocks, nous a permis de mettre à l'épreuve la méthodologie de modélisation SysML dans le but d'effectuer des simulations de convoyeur sans contact (jets d'air). Le second cas d'étude, issu du projet Région GEOSEFA, nous a permis de valider l'approche complète (simulation et test) en contexte In-the-Loop. Celui-ci porte sur la conception et la validation d'un nouveau système énergétique hybride embarqué dans un hélicoptère. / The research conducted during this thesis fall within the scope of modeling, verification and validation of critical and complex systems. This work aims to bridge the gap between the abstract high-level models, starting point of the MBSE process (Model-Based Systems Engineering), and real-time simulation keystone of In-the-Loop processes. In this context, we propose to unify, within a SysML model, continuous aspects of a system, to automatically generate an executable Modelica model (simulation), and discrete aspects allowing animation and test generation by constraint solvers. The work done during this thesis allowed the study and the realization of an original tooled approach to simulate and test such systems from SysML models within a In-the-Loop context. This approach has been validated by two concrete case studies from research partners. The first, from the ANR Smart Blocks project, allowed us to assess the relevance of the proposed SysML modeling methodology in order to perform contact less conveyor simulations. The second case study, from the GEOSEFA Regional project has allowed us to validate the overall approach (simulation and testing) in a In-the-Loop context. It covers the design and the validation of a new energy hybrid system embedded in a helicopter.

This master thesis investigates how well suited the model-based testing process is for testing a new feature of a Base Station Controller. In model-based testing the tester designs a behavioral model of the system under test, or some part of the system. This model is then given to a test generation tool that will analyze the model and produce interesting test cases. These test cases can either be run on the system in an automatic or manual way depending on what type of setup there is. In this report it is suggested that the behavioral model should be produced in as early a stage as possible and that it should be a collaboration between the test team and the design team. The advantages with the model-based testing process are a better overview of the test cases, the test cases are always up to date, it helps in finding errors or contradictions in requirements and it performs closer collaboration between the test team and the design team. The disadvantages with model-based testing process are that it introduces more sources where an error can occur. The behavioral model can have errors, the layer between the model and the generated test cases can have errors and the layer between the test cases and the system under test can have errors. This report also indicates that the time needed for testing will be longer compared with manual testing. During the pilot, when a part of a new feature was tested, of this master thesis a test generation tool called Qtronic was used. This tool solves a very challenging task which is generating test cases from a general behavioral model and with a good result. This tool provides many good things but it also has its shortages. One of the biggest shortages is the debugging of the model for finding errors. This step is very time consuming because it requires that a test case generation is performed on the whole model. When there is a fault in the model then this test generation can take very long time, before the tool decides that it is impossible to cover the model. Under the circumstances that the Qtronic tool is improved on varies issues suggested in the thesis, one of the most important issues is to do something about the long debugging time needed, then the next step can be to use model-based testing in a larger evaluation project at BSC Design, Ericsson.

Software testing is an important process for ensuring the quality of the software. As the complexity of the software increases, traditional means of manual testing becomes increasingly more complex and time consuming. In most embedded systems, designing software with as few errors as possible is often critical. Resource usage is also of concern for proper behavior because of the very nature of embedded systems.  To design reliable and energy-efficient systems, methods are needed to detect hot points of consumption and correct them prior to deployment. To reduce testing effort, Model-based testing can be used which is one testing method that allows for automatic testing of model based systems. Model-based testing has not been investigated extensively for revealing resource usage anomalies in embedded systems. UPPAAL SMC is a statistical model checking tool which can be used to model the system’s resource usage. Currently UPPAAL SMC lacks the support for performing automatic test generation and test selection. In this thesis we provide this support with a framework for automatic test generation and test selection using mutation analysis, a method for minimizing the generated test suite while maximizing the fault coverage and a tool implementing the framework on top of the UPPAAL SMC tool. The thesis also evaluates the framework on a Brake by Wire industrial system. Our results show that we could for a Brake-by-wire system, simulated on a consumer processor with five mutants, in best case find a test case that achieved 100% mutation score within one minute and confidently identify at least one test case that achieved full mutation score within five minutes. The evaluation shows that this framework is applicable and relatively efficient on an industrial system for reducing continues resource usage target testing effort.

Test and Diagnosis are critical areas in semiconductor manufacturing. Every chip manufactured using a new or premature technology or process needs to be tested for manufacturing defects to ensure defective chips are not sold to the customer. Conventionally, test is done by mounting the chip on an Automated Test Equipment (ATE) and applying test patterns to test for different faults. With shrinking feature sizes, the complexity of the circuits on chip is increasing, which in turn increases the number of test patterns needed to test the chip comprehensively. This increases the test application time which further increases the cost of test, ultimately leading to increase in the cost per device. Furthermore, chips that fail during test need to be diagnosed to determine the cause of the failure so that the manufacturing process can be improved to increase the yield. With increase in the size and complexity of the circuits, diagnosis is becoming an even more challenging and time consuming process. Fast diagnosis of failing chips can help in reducing the ramp-up to the high volume manufacturing stage and thus reduce the time to market. To reduce the time needed for diagnosis, efficient diagnostic patterns have to be generated that can distinguish between several faults. However, in order to reduce the test application time, the total number of patterns should be minimized. We propose a technique for generating diagnostic patterns that are inherently compact. Experimental results show up to 73% reduction in the number of diagnostic patterns needed to distinguish all faults. Logic Built-in Self-Test (LBIST) is an alternative methodology for testing, wherein all components needed to test the chip are on the chip itself. This eliminates the need of expensive ATEs and allows for at-speed testing of chips. However, there is hardware overhead incurred in storing deterministic test patterns on chip and failing chips are hard to diagnose in this LBIST architecture due to limited observability. We propose a technique to reduce the number of patterns needed to be stored on chip and thus reduce the hardware overhead. We also propose a new LBIST architecture which increases the diagnosability in LBIST with a minimal hardware overhead. These two techniques overcome the disadvantages of LBIST and can make LBIST more popular solution for testing of chips. Modern designs may contain a large number of small embedded memories. Memory Built-in Self-Test (MBIST) is the conventional technique of testing memories, but it incurs hardware overhead. Using MBIST for small embedded memories is impractical as the hardware overhead would be significantly high. Test generation for such circuits is difficult because the fault effect needs to be propagated through the memory. We propose a new technique for testing of circuits with embedded memories. By using SMT solver, we model memory at a high level of abstraction using theory of array, while keeping the surrounding logic at gate level. This effectively converts the test generation problem into a combinational test generation problem and make test generation easier than the conventional techniques. / Ph. D.

With the increasing complexity of modern day processors and system-on-a-chip (SOCs), designers invest a lot of time and resources into testing and validating these designs. To reduce the time-to-market and cost, the techniques used to validate these designs have to constantly improve. Since most of the design activity has moved to the register transfer level (RTL), test methodologies at the RTL have been gaining momentum. We present a novel functional test generation framework for functional test generation at RTL. A popular software-based metric for measuring the effectiveness of an RTL test suite is branch coverage. But exercising hard-to-reach branches is still a challenge and requires good understanding of the design semantics. The proposed framework uses static analysis to extract certain semantics of the circuit and uses several data structures to model these semantics. Using these data structures, we assist the branch-guided search to exercise these hard-to-reach branches. Since the correlation between high branch coverage and detecting defects and bugs is not clear, we present a new metric at the RTL which augments the RTL branch coverage with state values. Vectors which have higher scores on the new metric achieve higher branch and state coverages, and therefore can be applied at different levels of abstraction such as post-silicon validation. Experimental results show that use of the new metric in our test generation framework can achieve a high level of branch and fault coverage for several benchmark circuits, while reducing the length of the vector sequence. This work was supported in part by the NSF grant 1016675. / Master of Science