Submitted by Lucienne Costa (lucienneferreira@ufcg.edu.br) on 2018-08-30T18:41:36Z No. of bitstreams: 1 INDY PAULA SOARES CORDEIRO E SILVA – DISSERTAÇÃO (PPGCC) 2018.pdf: 1399246 bytes, checksum: ca2d8bfa9c9492d9d2d97a1485afad82 (MD5) / Made available in DSpace on 2018-08-30T18:41:36Z (GMT). No. of bitstreams: 1 INDY PAULA SOARES CORDEIRO E SILVA – DISSERTAÇÃO (PPGCC) 2018.pdf: 1399246 bytes, checksum: ca2d8bfa9c9492d9d2d97a1485afad82 (MD5) Previous issue date: 2018-02-22 / Capes / Refatoramentos normalmente exigem testes de regress˜ao para verificar se as mudanc¸as aplicadas ao c´odigo, preservaram o comportamento original do mesmo. Geralmente, ´e dif´ıcil definir um conjunto de testes que seja efetivo para esta tarefa, uma vez que o refatoramento n˜ao ´e frequentemente aplicado em etapas isoladas. Al´em disso, as edic¸ ˜oes de refatoramento podem ser combinadas com outras edic¸ ˜oes no c´odigo. Nesse sentido, a gerac¸ ˜ao de casos de teste pode contribuir para essa tarefa, analisando sistematicamente o c´odigo e fornecendo uma ampla gama de casos de teste que abordam diferentes construc¸ ˜oes. No entanto, uma s´erie de estudos apresentados na literatura mostram que as ferramentas atuais ainda n˜ao s˜ao eficazes com relac¸ ˜ao `a detecc¸ ˜ao de faltas, particularmente faltas de refatoramento. Com base nisto, apresentamos dois estudos emp´ıricos que aplicaram as ferramentas Randoop e Evosuite para gerar suites de testes de regress˜ao, com foco na edic¸ ˜ao de refatoramento do tipo extract method. Com base nos resultados dos estudos, identificamos fatores que podem influenciar o desempenho das ferramentas para efetivamente testar a edic¸ ˜ao. Para validar nossos achados, apresentamos um conjunto de modelos de regress˜ao que associam a presenc¸a desses fatores `a capacidade do conjunto de testes, de detectar faltas relacionadas `a edic¸ ˜ao de refatoramento. E por fim, apresentamos a REFANALYZER, que ´e uma ferramenta que objetiva ajudar os desenvolvedores a decidir quando confiar em suites geradas automaticamente, para validar refatoramento do tipo extract method. / Refactoring typically require regression testers to verify that the changes applied to the code have preserved the original behavior of the code. Generally, it is difficult to define a set of tests that is effective for this task, since refactoring is not the weight applied in isolated steps. Also, since refactoring issues can be combined with other issues, without code. In this sense, a generation of test cases can contribute to this task by systematically analyzing the code and defining a wide range of test cases that address different constructs. However, a number of studies are not available but are not very effective in detecting faults, particularly refactorings. Based on this, we present two empirical studies that have applied as Randoop and Evosuite tools to generate regression test suites, focusing on the edition of refactoring of the extract method type. Based on the results of the studies, we identified factors that can influence the performance of the tools to effectively test the edition. To validate our findings, we present a set of control models that associate a solution with the ability of the set of tests, of spoken faults related to the refactoring edition. And finally, we present a REFANALYZER, which is a tool that is a solution to what needs to be solved when it is automatically managed, to validate refactoring of the extract method type.

A utilização desta abordagem no ciclo de desenvolvimento de uma aplicação web traz algumas vantagens, como ser necessário gerar manualmente apenas o modelo de comportamento de cada funcionalidade da aplicação, (pois os demais artefatos são gerados automaticamente), consumindo menos tempo e estando menos sujeitos a erros, além de prevenir diferentes interpretações dos requisitos pelos stakeholders, desenvolvedores e testadores. O tempo despendido na especificação dos modelos é compensado pelo tempo economizado com a geração dos cenários e do código de testes. / The testing activity throughout software development is fundamental to the pursuit of software quality and reliability, finding faults to be removed. However, despite its importance, software testing is often an underutilized phase in software development. Moreover, tests are proved to be expensive, difficult and problematic when not done in the appropriate way. A new paradigm for software testing is model-driven testing (MDT), which can be defined as software testing where test cases are derived from a model that describes some aspects of the system being tested, such as behavior, for example. This description, often using UML diagrams and/or its profiles, can be processed to produce a set of test cases. Software specifications based on usage scenarios expressed by appropriate UML diagrams are considered significant and effective, because they describe the system’s requirements from an intuitive and visual perspective. Thus, they can be used for the description of acceptance tests, which validate that the system meets user requirements. These specifications also facilitate the automation of this kind of test. Test automation can decrease time spent on testing, thereby reducing the cost of this activity. Thus, this work proposes an approach for automated generation of acceptance tests from U2TP (the UML 2.0 test profile) diagrams for web applications, based on behavior driven development (BDD) paradigm, obtaining acceptance scenarios and executable test code supported by an acceptance testing automation framework. This approach was applied on an actual development environment, by means of an experiment. Using this approach in an web application development cycle has some advantages, such as being required only to manually generate the model of behavior of each application functionality (because other artifacts are generated automatically), thus being less time consuming and less prone to errors, and preventing different interpretations of requirements by stakeholders, developers and testers. The time spent at the models’ specification is compensated by the time saved with the generation of scenarios and test code.

A utilização desta abordagem no ciclo de desenvolvimento de uma aplicação web traz algumas vantagens, como ser necessário gerar manualmente apenas o modelo de comportamento de cada funcionalidade da aplicação, (pois os demais artefatos são gerados automaticamente), consumindo menos tempo e estando menos sujeitos a erros, além de prevenir diferentes interpretações dos requisitos pelos stakeholders, desenvolvedores e testadores. O tempo despendido na especificação dos modelos é compensado pelo tempo economizado com a geração dos cenários e do código de testes. / The testing activity throughout software development is fundamental to the pursuit of software quality and reliability, finding faults to be removed. However, despite its importance, software testing is often an underutilized phase in software development. Moreover, tests are proved to be expensive, difficult and problematic when not done in the appropriate way. A new paradigm for software testing is model-driven testing (MDT), which can be defined as software testing where test cases are derived from a model that describes some aspects of the system being tested, such as behavior, for example. This description, often using UML diagrams and/or its profiles, can be processed to produce a set of test cases. Software specifications based on usage scenarios expressed by appropriate UML diagrams are considered significant and effective, because they describe the system’s requirements from an intuitive and visual perspective. Thus, they can be used for the description of acceptance tests, which validate that the system meets user requirements. These specifications also facilitate the automation of this kind of test. Test automation can decrease time spent on testing, thereby reducing the cost of this activity. Thus, this work proposes an approach for automated generation of acceptance tests from U2TP (the UML 2.0 test profile) diagrams for web applications, based on behavior driven development (BDD) paradigm, obtaining acceptance scenarios and executable test code supported by an acceptance testing automation framework. This approach was applied on an actual development environment, by means of an experiment. Using this approach in an web application development cycle has some advantages, such as being required only to manually generate the model of behavior of each application functionality (because other artifacts are generated automatically), thus being less time consuming and less prone to errors, and preventing different interpretations of requirements by stakeholders, developers and testers. The time spent at the models’ specification is compensated by the time saved with the generation of scenarios and test code.

A utilização desta abordagem no ciclo de desenvolvimento de uma aplicação web traz algumas vantagens, como ser necessário gerar manualmente apenas o modelo de comportamento de cada funcionalidade da aplicação, (pois os demais artefatos são gerados automaticamente), consumindo menos tempo e estando menos sujeitos a erros, além de prevenir diferentes interpretações dos requisitos pelos stakeholders, desenvolvedores e testadores. O tempo despendido na especificação dos modelos é compensado pelo tempo economizado com a geração dos cenários e do código de testes. / The testing activity throughout software development is fundamental to the pursuit of software quality and reliability, finding faults to be removed. However, despite its importance, software testing is often an underutilized phase in software development. Moreover, tests are proved to be expensive, difficult and problematic when not done in the appropriate way. A new paradigm for software testing is model-driven testing (MDT), which can be defined as software testing where test cases are derived from a model that describes some aspects of the system being tested, such as behavior, for example. This description, often using UML diagrams and/or its profiles, can be processed to produce a set of test cases. Software specifications based on usage scenarios expressed by appropriate UML diagrams are considered significant and effective, because they describe the system’s requirements from an intuitive and visual perspective. Thus, they can be used for the description of acceptance tests, which validate that the system meets user requirements. These specifications also facilitate the automation of this kind of test. Test automation can decrease time spent on testing, thereby reducing the cost of this activity. Thus, this work proposes an approach for automated generation of acceptance tests from U2TP (the UML 2.0 test profile) diagrams for web applications, based on behavior driven development (BDD) paradigm, obtaining acceptance scenarios and executable test code supported by an acceptance testing automation framework. This approach was applied on an actual development environment, by means of an experiment. Using this approach in an web application development cycle has some advantages, such as being required only to manually generate the model of behavior of each application functionality (because other artifacts are generated automatically), thus being less time consuming and less prone to errors, and preventing different interpretations of requirements by stakeholders, developers and testers. The time spent at the models’ specification is compensated by the time saved with the generation of scenarios and test code.

Software testing is an important process for ensuring the quality of the software. As the complexity of the software increases, traditional means of manual testing becomes increasingly more complex and time consuming. In most embedded systems, designing software with as few errors as possible is often critical. Resource usage is also of concern for proper behavior because of the very nature of embedded systems.  To design reliable and energy-efficient systems, methods are needed to detect hot points of consumption and correct them prior to deployment. To reduce testing effort, Model-based testing can be used which is one testing method that allows for automatic testing of model based systems. Model-based testing has not been investigated extensively for revealing resource usage anomalies in embedded systems. UPPAAL SMC is a statistical model checking tool which can be used to model the system’s resource usage. Currently UPPAAL SMC lacks the support for performing automatic test generation and test selection. In this thesis we provide this support with a framework for automatic test generation and test selection using mutation analysis, a method for minimizing the generated test suite while maximizing the fault coverage and a tool implementing the framework on top of the UPPAAL SMC tool. The thesis also evaluates the framework on a Brake by Wire industrial system. Our results show that we could for a Brake-by-wire system, simulated on a consumer processor with five mutants, in best case find a test case that achieved 100% mutation score within one minute and confidently identify at least one test case that achieved full mutation score within five minutes. The evaluation shows that this framework is applicable and relatively efficient on an industrial system for reducing continues resource usage target testing effort.

Les travaux de recherche menés au cours de cette thèse s'inscrivent dans le cadre de la modélisation, de la vérification et de la validation de systèmes complexes, critiques et multi-physiques. Ces travaux visent à combler l'écart d'abstraction entre les modèles haut-niveau, point de départ des processus MBSE (Model-Based Systems Engineering), et la simulation temps réel, clef de voûte des approches In-the-Loop. Dans ce contexte, nous proposons d'unifier, au sein d'un même modèle SysML, les aspects continus d'un système, permettant de générer de manière automatique un modèle Modelica de plus bas niveau directement exécutable (simulation), et les aspects discrets, permettant l'animation et la génération de tests par des solveurs de contraintes. Les travaux réalisés au cours de cette thèse ont permis l'étude et la réalisation d'une chaîne outillée originale permettant de simuler et de tester ce type de systèmes à partir de modèles SysML en contexte In-the-Loop. Cette démarche a été validée par deux cas d'étude concrets issus de la recherche. Le premier, issu du projet ANR Smart Blocks, nous a permis de mettre à l'épreuve la méthodologie de modélisation SysML dans le but d'effectuer des simulations de convoyeur sans contact (jets d'air). Le second cas d'étude, issu du projet Région GEOSEFA, nous a permis de valider l'approche complète (simulation et test) en contexte In-the-Loop. Celui-ci porte sur la conception et la validation d'un nouveau système énergétique hybride embarqué dans un hélicoptère. / The research conducted during this thesis fall within the scope of modeling, verification and validation of critical and complex systems. This work aims to bridge the gap between the abstract high-level models, starting point of the MBSE process (Model-Based Systems Engineering), and real-time simulation keystone of In-the-Loop processes. In this context, we propose to unify, within a SysML model, continuous aspects of a system, to automatically generate an executable Modelica model (simulation), and discrete aspects allowing animation and test generation by constraint solvers. The work done during this thesis allowed the study and the realization of an original tooled approach to simulate and test such systems from SysML models within a In-the-Loop context. This approach has been validated by two concrete case studies from research partners. The first, from the ANR Smart Blocks project, allowed us to assess the relevance of the proposed SysML modeling methodology in order to perform contact less conveyor simulations. The second case study, from the GEOSEFA Regional project has allowed us to validate the overall approach (simulation and testing) in a In-the-Loop context. It covers the design and the validation of a new energy hybrid system embedded in a helicopter.

This master thesis investigates how well suited the model-based testing process is for testing a new feature of a Base Station Controller. In model-based testing the tester designs a behavioral model of the system under test, or some part of the system. This model is then given to a test generation tool that will analyze the model and produce interesting test cases. These test cases can either be run on the system in an automatic or manual way depending on what type of setup there is. In this report it is suggested that the behavioral model should be produced in as early a stage as possible and that it should be a collaboration between the test team and the design team. The advantages with the model-based testing process are a better overview of the test cases, the test cases are always up to date, it helps in finding errors or contradictions in requirements and it performs closer collaboration between the test team and the design team. The disadvantages with model-based testing process are that it introduces more sources where an error can occur. The behavioral model can have errors, the layer between the model and the generated test cases can have errors and the layer between the test cases and the system under test can have errors. This report also indicates that the time needed for testing will be longer compared with manual testing. During the pilot, when a part of a new feature was tested, of this master thesis a test generation tool called Qtronic was used. This tool solves a very challenging task which is generating test cases from a general behavioral model and with a good result. This tool provides many good things but it also has its shortages. One of the biggest shortages is the debugging of the model for finding errors. This step is very time consuming because it requires that a test case generation is performed on the whole model. When there is a fault in the model then this test generation can take very long time, before the tool decides that it is impossible to cover the model. Under the circumstances that the Qtronic tool is improved on varies issues suggested in the thesis, one of the most important issues is to do something about the long debugging time needed, then the next step can be to use model-based testing in a larger evaluation project at BSC Design, Ericsson.

Software testing is an important process for ensuring the quality of the software. As the complexity of the software increases, traditional means of manual testing becomes increasingly more complex and time consuming. In most embedded systems, designing software with as few errors as possible is often critical. Resource usage is also of concern for proper behavior because of the very nature of embedded systems.  To design reliable and energy-efficient systems, methods are needed to detect hot points of consumption and correct them prior to deployment. To reduce testing effort, Model-based testing can be used which is one testing method that allows for automatic testing of model based systems. Model-based testing has not been investigated extensively for revealing resource usage anomalies in embedded systems. UPPAAL SMC is a statistical model checking tool which can be used to model the system’s resource usage. Currently UPPAAL SMC lacks the support for performing automatic test generation and test selection. In this thesis we provide this support with a framework for automatic test generation and test selection using mutation analysis, a method for minimizing the generated test suite while maximizing the fault coverage and a tool implementing the framework on top of the UPPAAL SMC tool. The thesis also evaluates the framework on a Brake by Wire industrial system. Our results show that we could for a Brake-by-wire system, simulated on a consumer processor with five mutants, in best case find a test case that achieved 100% mutation score within one minute and confidently identify at least one test case that achieved full mutation score within five minutes. The evaluation shows that this framework is applicable and relatively efficient on an industrial system for reducing continues resource usage target testing effort.

The Java platform and third-party open-source libraries provide various Application Programming Interfaces (APIs) to facilitate secure coding. However, using these APIs securely is challenging for developers who lack cybersecurity training. Prior studies show that many developers use APIs insecurely, thereby introducing vulnerabilities in their software. Despite the availability of various tools designed to identify API insecure usage, their effectiveness in helping developers with secure coding practices remains unclear. This dissertation focuses on two main objectives: (1) exploring the strengths and weaknesses of the existing automated detection tools for API-related vulnerabilities, and (2) creating better tools that detect, repair, and demonstrate these vulnerabilities. Our research started with investigating the effectiveness of current tools in helping with developers' secure coding practices. We systematically explored the strengths and weaknesses of existing automated tools for detecting API-related vulnerabilities. Through comprehensive analysis, we observed that most existing tools merely report misuses, without suggesting any customized fixes. Moreover, developers often rejected tool-generated vulnerability reports due to their concerns on the correctness of detection, and the exploitability of the reported issues. To address these limitations, the second work proposed SEADER, an example-based approach to detect and repair security-API misuses. Given an exemplar ⟨insecure, secure⟩ code pair, SEADER compares the snippets to infer any API-misuse template and corresponding fixing edit. Based on the inferred information, given a program, SEADER performs inter-procedural static analysis to search for security-API misuses and to propose customized fixes. The third work leverages ChatGPT-4.0 to automatically generate security test cases. These test cases can demonstrate how vulnerable API usage facilitates supply chain attacks on specific software applications. By running such test cases during software development and maintenance, developers can gain more relevant information about exposed vulnerabilities, and may better create secure-by-design and secure-by-default software. / Doctor of Philosophy / The Java platform and third-party open-source libraries provide various Application Pro- gramming Interfaces (APIs) to facilitate secure coding. However, using these APIs securely can be challenging, especially for developers who aren't trained in cybersecurity. Prior work shows that many developers use APIs insecurely, consequently introducing vulnerabilities in their software. Despite the availability of various tools designed to identify API insecure usage, it is still unclear how well they help developers with secure coding practices. This dissertation focuses on (1) exploring the strengths and weaknesses of the existing au- tomated detection tools for API-related vulnerabilities, and (2) creating better tools that detect, repair, and demonstrate these vulnerabilities. We first systematically evaluated the strengths and weaknesses of the existing automated API-related vulnerability detection tools. We observed that most existing tools merely report misuses, without suggesting any cus- tomized fixes. Additionally, developers often reject tool-generated vulnerability reports due to their concerns about the correctness of detection, and whether the reported vulnerabil- ities are truly exploitable. To address the limitations found in our study, the second work proposed a novel example-based approach, SEADER, to detect and repair API insecure usage. The third work leverages ChatGPT-4.0 to automatically generate security test cases, and to demonstrate how vulnerable API usage facilitates the supply chain attacks to given software applications.

The increasing design complexity and shrinking feature size of hardware designs have created resource intensive design verification and manufacturing test phases in the product life-cycle of a digital system. On the contrary, time-to-market constraints require faster verification and test phases; otherwise it may result in a buggy design or a defective product. This trend in the semiconductor industry has considerably increased the complexity and importance of Design Verification, Manufacturing Test and Silicon Diagnosis phases of a digital system production life-cycle. In this dissertation, we present a generalized learning framework, which can be customized to the common solving technique for problems in these three phases. During Design Verification, the conformance of the final design to its specifications is verified. Simulation-based and Formal verification are the two widely known techniques for design verification. Although the former technique can increase confidence in the design, only the latter can ensure the correctness of a design with respect to a given specification. Originally, Design Verification techniques were based on Binary Decision Diagram (BDD) but now such techniques are based on branch-and-bound procedures to avoid space explosion. However, branch-and-bound procedures may explode in time; thus efficient heuristics and intelligent learning techniques are essential. In this dissertation, we propose a novel extensibility relation between search states and a learning framework that aids in identifying non-trivial redundant search states during the branch-and-bound search procedure. Further, we also propose a probability based heuristic to guide our learning technique. First, we utilize this framework in a branch-and-bound based preimage computation engine. Next, we show that it can be used to perform an upper-approximation based state space traversal, which is essential to handle industrial-scale hardware designs. Finally, we propose a simple but elegant image extraction technique that utilizes our learning framework to compute over-approximate image space. This image computation is later leveraged to create an abstraction-refinement based model checking framework. During Manufacturing Test, test patterns are applied to the fabricated system, in a test environment, to check for the existence of fabrication defects. Such patterns are usually generated by Automatic Test Pattern Generation (ATPG) techniques, which assume certain fault types to model arbitrary defects. The size of fault list and test set has a major impact on the economics of manufacturing test. Towards this end, we propose a fault col lapsing approach to compact the size of target fault list for ATPG techniques. Further, from the very beginning, ATPG techniques were based on branch-and-bound procedures that model the problem in a Boolean domain. However, ATPG is a problem in the multi-valued domain; thus we propose a multi-valued ATPG framework to utilize this underlying nature. We also employ our learning technique for branch-and-bound procedures in this multi-valued framework. To improve the yield for high-volume manufacturing, silicon diagnosis identifies a set of candidate defect locations in a faulty chip. Subsequently physical failure analysis - an extremely time consuming step - utilizes these candidates as an aid to locate the defects. To reduce the number of candidates returned to the physical failure analysis step, efficient diagnostic patterns are essential. Towards this objective, we propose an incremental framework that utilizes our learning technique for a branch-and-bound procedure. Further, it learns from the ATPG phase where detection-patterns are generated and utilizes this information during diagnostic-pattern generation. Finally, we present a probability based heuristic for X-filling of detection-patterns with the objective of enhancing the diagnostic resolution of such patterns. We unify these techniques into a framework for test pattern generation with good detection and diagnostic ability. Overall, we propose a learning framework that can speed up design verification, test and diagnosis steps in the life cycle of a hardware system. / Ph. D.