O teste de mutação é um critério de teste poderoso para detectar falhas e medir a eficácia de um conjunto de dados de teste. No entanto, é uma técnica de teste computacionalmente cara. O alto custo provém principalmente do esforço para gerar dados de teste adequados para matar os mutantes e pela existência de mutantes equivalentes. Nesse contexto, o objetivo desta tese é apresentar uma abordagem chamada de Reach, Infect and Propagation to Mutation Testing (RIPMuT) que visa gerar dados de teste e sugerir mutantes equivalentes. A abordagem é composta por dois módulos: (i) uma geração automatizada de dados de teste usando subida da encosta e um esquema de fitness de acordo com as condições de alcançabilidade, infeção e propagação (RIP); e (ii) um método para sugerir mutantes equivalentes com base na análise das condições RIP durante o processo de geração de dados de teste. Os experimentos foram conduzidos para avaliar a eficácia da abordagem RIP-MuT e um estudo comparativo com o algoritmo genético e testes aleatórios foi realizado. A abordagem RIP-MuT obteve um escore médio de mutação de 18,25 % maior que o AG e 35,93 % maior que o teste aleatório. O método proposto para detecção de mutantes equivalentes se mostrou viável para redução de custos relacionado a essa atividade, uma vez que obteve uma precisão de 75,05% na sugestão dos mutantes equivalentes. Portanto, os resultados indicam que a abordagem gera dados de teste adequados capazes de matar a maioria dos mutantes em programas C e, também auxilia a identificar mutantes equivalentes corretamente. / Mutation Testing is a powerful test criterion to detect faults and measure the effectiveness of a test data set. However, it is a computationally expensive testing technique. The high cost comes mainly from the effort to generate adequate test data to kill the mutants and by the existence of equivalent mutants. In this thesis, an approach called Reach, Infect and Propagation to Mutation Testing (RIP-MuT) is presented to generate test data and to suggest equivalent mutants. The approach is composed of two modules: (i) an automated test data generation using hill climbing and a fitness scheme according to Reach, Infect, and Propagate (RIP) conditions; and (ii) a method to suggest equivalent mutants based on the analyses of RIP conditions during the process of test data generation. The experiments were conducted to evaluate the effectiveness of the RIP-MuT approach and a comparative study with a genetic algorithm and random testing. The RIP-MuT approach achieved a mean mutation score of 18.25% higher than the GA and 35.93% higher than random testing. The proposed method for detection of equivalent mutants demonstrate to be feasible for cost reduction in this activity since it obtained a precision of 75.05% on suggesting equivalent mutants. Therefore, the results indicate that the approach produces effective test data able to strongly kill the majority of mutants on C programs, and also it can assist in suggesting equivalent mutants correctly.

Submitted by Natalia de Souza Gonçalves (natalia.goncalves@ufpe.br) on 2016-09-28T12:23:59Z No. of bitstreams: 2 license_rdf: 1232 bytes, checksum: 66e71c371cc565284e70f40736c94386 (MD5) Dissertacao_Cleydiane_FINAL_Imprimir.pdf: 3971527 bytes, checksum: f06a014d16a761f030bbfccee75d176f (MD5) / Made available in DSpace on 2016-09-28T12:23:59Z (GMT). No. of bitstreams: 2 license_rdf: 1232 bytes, checksum: 66e71c371cc565284e70f40736c94386 (MD5) Dissertacao_Cleydiane_FINAL_Imprimir.pdf: 3971527 bytes, checksum: f06a014d16a761f030bbfccee75d176f (MD5) Previous issue date: 2016-02-19 / Dívida Técnica (DT) está relacionada a tarefas que devem ser executadas e são acumuladas ao longo de um projeto para serem realizadas posteriormente. A falta de planejamento de tempo e recursos em um projeto, dentre outros problemas, propiciam seu surgimento. DTs não solucionadas tendem a criar dependências e aumentar o grau de complexidade para correção, resultando em maior esforço, custo e retrabalho em um projeto. A sua identificação e gerenciamento ao longo de um projeto é essencial para minimizar seu impacto e consequências negativas. Estudos apontam teste de software como uma das principais áreas impactadas por DTs em projetos de software. Nesta dissertação foram mapeadas, a partir da literatura técnica, 22 possíveis DTs relacionadas ao processo de teste de software, assim como suas causas e indicadores. Elas foram avaliadas por meio de um survey com profissionais da área de teste de software. Como resultados do survey, são apresentados os níveis de concordância obtidos para cada DT, suas causas e indicadores sugeridos. Após essa validação da concordância quanto às DTs, foi identificada a necessidade de vincular soluções às causas de cada DT, para isto foi realizada uma nova revisão de literatura. A partir dos dados das DTs, com suas causas, indicadores e, agora, soluções, foi elaborado um mapa, com objetivo de apoiar profissionais da área de teste de software na gestão de DTs que podem ocorrer no decorrer da execução do processo de teste de software. Após a elaboração do mapa de gestão de dívida técnica, o mesmo foi avaliado a partir de um questionário (baseado na técnica de revisão em pares), por profissionais experientes em teste de software (líderes e consultores). Como resultado desta avaliação, são apresentadas as opiniões e considerações destes profissionais sobre a versão inicial do mapa elaborado. Por fim, é apresentado o resultado final do mapa, com as devidas alterações baseadas nas respostas obtidas no questionário de avaliação do mapa. / Technical Debit (TD) is related to the tasks which have to be executed but are accumulated during the development of a project to be implemented later. The lack of time and resource planning on a project, among other problems, facilitate its emergence. Unsolved TDs tend to create dependencies and increase the degree of complexity for corrections, resulting in bigger efforts, costs and rework on a project. Identifying and managing those issues are essential to minimize the impacts and negative consequences. Studies point out software testing as one of the most affected areas by TDs on a software project. At this dissertation, 22 (twenty-two) possible TDs related to testing software process were mapped from the technical lecture, along with its causes and indicators. They were evaluated through surveys with testing software experts. As result of the survey, it is presented the levels of accordance for each TD, also the suggested causes and indicators. After the accordance validation of the TDs, it was identified a demand to bind solutions to the causes of every TD, for this a new study was made. From the data of the TDs, along with its causes, indicators and solutions, a map was produced in order to support testing software professionals managing TDs that may appear during the testing software process execution. After producing the technical debit management map, the same was submitted for an evaluation, using a questionnaire (based on peer review technique), to testing software experts (leaders and consultants). As result of these evaluations, it is presented the opinions and remarks of those professionals about the final version of the produced map. Finally, it is presented the final result of the map, with the modifications based on the answers obtained through the map evaluation questionnaire.

Context: Software testing is the process of assessing quality of a software product to determine whether it matches with the existing requirements of the customer or not. Software testing is one of the “Verification and Validation,” or V&V, software practices. The two basic techniques of software testing are Black-box testing and White box testing. Black-box testing focuses solely on the outputs generated in response to the inputs supplied neglecting the internal components of the software. Whereas, White-box testing focuses on the internal mechanism of the software of any application. To explore the feasibility of black-box and white-box testing under a given set of conditions, a proper test automation framework needs to be deployed. Automation is deployed in order to reduce the manual effort and to perform testing continuously, thereby increasing the quality of the product.   Objectives: In this research, cloud hosted application is automated using TestComplete tool. The objective of this thesis is to verify the functionality of Cloud application known as Test data library or Test Report Analyzer through automation and to measure the impact of the automation on release cycles of the organization.   Methods: Here automation is implemented using scrum methodology which is an agile development software process. Using scrum methodology, the product with working software can be delivered to the customers incrementally and empirically with updating functionalities in it. Test data library or Test Report Analyzer functionality of Cloud application is verified deploying testing device thereby the test cases can be analyzed thereby analyzing the pass or failed test cases.   Results: Automation of test report analyzer functionality of cloud hosted application is made using TestComplete and impact of automation on release cycles is reduced. Using automation, nearly 24% of change in release cycles can be observed thereby reducing the manual effort and increasing the quality of delivery.   Conclusion: Automation of a cloud hosted application provides no manual effort thereby utilization of time can be made effectively and application can be tested continuously increasing the efficiency and the quality of an application. / AUTOMATION OF A CLOUD HOSTED APPLICATION

Many challenges in software quality can be tackled with dynamic analysis. However, these techniques are often limited in their efficiency or scalability as they are often applied uniformly to an entire program. In this thesis, we show that dynamic program analysis can be made significantly more efficient and scalable by first performing a static data flow analysis so that the dynamic analysis can be selectively applied only to important parts of the program. We apply this general principle to the design and implementation of two different systems, one for runtime security policy enforcement and the other for software test input generation. For runtime security policy enforcement, we enforce user-defined policies using a dynamic data flow analysis that is more general and flexible than previous systems. Our system uses the user-defined policy to drive a static data flow analysis that identifies and instruments only the statements that may be involved in a security vulnerability, often eliminating the need to track most objects and greatly reducing the overhead. For taint analysis on a set of five server programs, the slowdown is only 0.65%, two orders of magnitude lower than previous taint tracking systems. Our system also has negligible overhead on file disclosure vulnerabilities, a problem that taint tracking cannot handle. For software test case generation, we introduce the idea of targeted testing, which focuses testing effort on select parts of the program instead of treating all program paths equally. Our “Bullseye” system uses a static analysis performed with respect to user-defined “interesting points” to steer the search down certain paths, thereby finding bugs faster. We also introduce a compiler transformation that allows symbolic execution to automatically perform boundary condition testing, revealing bugs that could be missed even if the correct path is tested. For our set of 9 benchmarks, Bullseye finds bugs an average of 2.5× faster than a conventional depth-first search and finds numerous bugs that DFS could not. In addition, our automated boundary condition testing transformation allows both Bullseye and depth-first search to find numerous bugs that they could not find before, even when all paths were explored. / text

Efficiency as a term in software testing is, in the research community, a term that is not so well defined. In the industry, and specifically the test tool industry, it has become a sales pitch without meaning. GUI testing in its manual form is a time consuming task, which can be thought of as repetitive and tedious by testers. Using human testers to perform a task, where focus is hard to keep, often ends in defects going unnoticed. The purpose of this thesis is to collect knowledge on the area efficiency in software testing, but focusing more on efficiency in GUI testing in order to keep the scope focused. Part of the purpose is also to test the hypothesis that automated GUI testing is more efficient than traditional, manual GUI testing. In order to reach the purpose, the choice fell to use case study research as the main research method. Through the case study, a theoretical study was performed to gain knowledge on the subject. To gain data used for an analysis in the case study, the choice fell on using a semi-experimental research approach where one automated GUI testing technique called Capture & Replay was tested against a more traditional approach towards GUI testing. The results obtained throughout the case study gives a definition on efficiency in software testing, as well as three measurements on efficiency, those being defect detection, repeatability of test cases, and time spent with human interaction. The result also includes the findings from the semi-experimental research approach where the testing tools Squish, and TestComplete, where used beside a manual testing approach. The main conclusion deducted in this work is that an automated approach towards GUI testing can become more efficient than a manual approach, in the long run. This is when efficiency is determined on the points of defect detection, repeatability, and time.

Introductory programming courses compose the core of several undergraduate degree programs, since programming is a crucial technical skill for professionals in many STEM areas. Aiming to address students learning difficulties in these courses, instructors can adopt different teaching approaches, since there are several varying aspects in programming education (e.g. programming languages and paradigms, development practices, platforms, supporting tools etc). In particular, the teaching approach that consists of integrating software testing into this context has been prominent in the area, since it may lead students to think more critically while working on programming assignments. Even so, this teaching approach can also present significant challenges, such as the students reluctance to conduct testing practices. In this sense, experimental studies have the role to provide evidence about learning outcomes, considering different teaching approaches and contexts. However, studies in the area of programming education often present a lack of theoretical basis, i.e. are not built upon established theories, models and frameworks. In other words, the varying aspects (or variables) used to investigate teaching approaches are not properly characterized in the studies, what leads to difficulties to interpret the obtained results and build knowledge in the area. As a consequence, instructors are prevented from having reliable evidence to make informed choices of teaching approaches used in the classroom. Considering this scenario, we propose in this PhD thesis the use of domain-specific models to support researchers in scoping and designing experiments on programming education. More specifically, we explored the domain of the software testing integration teaching approach, by creating a framework for experimental studies on the integration of software testing into programming education. The framework provides a basic structure of experimental studies in this domain, composed by models of variables related to this teaching approach. We also conducted experiments on the same domain and demonstrated their instantiation into the framework. We intend to support researchers and instructors in the scoping and planning of experimental studies in the educational scenario, specially those aimed at evaluating the integration of software testing into programming courses. / Disciplinas introdutórias de programação compõem o núcleo de diversos cursos de graduação, visto que se trata de uma habilidade crucial para profissionais em muitas áreas de ciências exatas. Buscando lidar com as dificuldades de aprendizagem dos alunos nessas disciplinas, os professores podem adotar diferentes abordagens de ensino, uma vez que há muitas variantes no ensino de programação (como linguagens e paradigmas de programação, práticas de desenvolvimento, plataformas, ferramentas de apoio, etc). Em particular, a abordagem de ensino que consiste em integrar teste de software nesse contexto tem se destacado na área, pois pode levar os alunos a pensarem de modo mais crítico enquanto resolvem atividades práticas de programação. Mesmo assim, essa abordagem de ensino também pode apresentar desafios significativos, como a resistência dos alunos para conduzir práticas de teste. Nesse sentido, estudos experimentais têm o papel de fornecer evidência acerca de resultados em termos de aprendizagem, considerando diferentes abordagens de ensino e contextos. Porém, estudos na área de ensino de programação muitas vezes apresentam falta de fundamentação teórica, ou seja, não são construídos a partir de teorias, modelos e frameworks estabelecidos na área. Isso significa que os aspectos (ou variáveis) utilizados para investigar as abordagens de ensino não são adequadamente caracterizados nos estudos, o que leva a dificuldades em interpretar os resultados obtidos e construir conhecimento na área. Como consequência, os professores são impedidos de ter evidências confiáveis para fazer escolhas informadas nas abordagens de ensino utilizadas em sala de aula. Considerando esse cenário, este trabalho de doutorado propõe o uso de modelos de domínio para apoiar pesquisadores ao definir e projetar experimentos no ensino de programação. Mais especificamente, o domínio da abordagem de integração de teste de software foi explorado neste trabalho, com a criação de um framework para estudos experimentais sobre a integração de teste de software no ensino de programação. O framework provê uma estrutura básica de estudos experimentais nesse domínio, sendo composto por modelos de variáveis relacionadas a essa abordagem de ensino. Neste trabalho também foram conduzidos experimentos de acordo com a estrutura do framework. A meta deste trabalho é apoiar pesquisadores e professores ao definir e planejar estudos no cenário educacional, em especial os focados em avaliar a integração de teste de software em disciplinas de programação.

Digital services involving large systems with multiple users are ubiquitous in modern society. The systems are often complicated and made up of multiple devices and communication protocols. A fundamental problem in this context is how the behavior of a system changes as the number of users vary. In particular, when do the systems’ resources saturate and how does the system behave when close to saturation. Performance testing is key for addressing this fundamental problem. Performance testing is the scope of this project. Performance tests can be used for inference of, for example, a system's scalability. Furthermore, it can be used to provide general guarantees on the services that can be delivered. Performance testing at the company Phoniro AB is considered. The platform Phoniro Care is the back-end service for the company’s products. The Phoiro 6000 system is one of the products that uses Phoniro Care. The system allows for multiple users and offers alarm services. The primary focus of this project is to determine the behavior of that system during varying levels of simulated load, and furthermore analyze the data extracted from such simulations and tests. The open source software JMeter was used as the tool for performance testing. It was selected from a set of candidate tools that have been evaluated in the literature based on various performance criteria. The results are presented by graphs showing the time evolution of different performance indicators. A conclusion from this work is that the implemented performance testing framework helps to answer questions about the systems’ behavior. Questions that are important for the company’s further development and expansion of the system. Furthermore, the proposed framework establishes a foundation for further inquiries on the subject.

No desenvolvimento de software, a qualidade do produto está diretamente relacionada à qualidade do processo de desenvolvimento. Diante disso, atividades de Verificação, Validação & Teste (VV&T) realizadas por meio de métodos, técnicas e ferramentas são de extrema necessidade para o aumento da produtividade, qualidade e diminuição de custos no desenvolvimento de software. Do mesmo modo, técnicas e critérios contribuem para a produtividade das atividades de teste. Um ponto crucial para o teste de software é sua automatização, tornando as atividades mais confiáveis e diminuindo significativamente os custos de desenvolvimento. Na automatização dos testes, os oráculos são essenciais, representando um mecanismo (programa, processo ou dados) que indica se a saída obtida para um caso de teste está correta. Este trabalho de mestrado utiliza a ideia de mutação para criar implementações alternativas de oráculos de teste e, assim, avaliar a sua qualidade. O teste de mutação se refere à criação de versões do sistema em desenvolvimento com pequenas alterações sintáticas de código. A mutação possui alta eficácia na detecção de defeitos e é bastante flexível na sua aplicação, podendo ser utilizada em diversos tipos de artefatos. Adicionalmente, este trabalho propõe operadores de mutação específicos para oráculos, implementa uma ferramenta de apoio à utilização desses operadores para oráculos e também descreve um estudo empírico dos operadores, destacando benefícios e desafios associados ao seu uso. / In software development, product quality is directly related to the quality of the development process. Therefore, activities of Verification, Validation & Testing (VV&T) performed by methods, techniques and tools are urgently required to increase productivity, quality and cost reduction in software development. Similarly, testing technique and criteria contribute to the productivity of test activities. A crucial point for the software testing automation is making the most reliable activities and significantly reducing development costs. Regarding software testing automation, test oracles are essential, representing an mechanism (program, process or data) to indicate whether the actual output for a given test case is correct. This masters thesis aims to explore concepts of mutation testing to create alternative implementations of the oracle procedure and thus assess their quality. Mutation testing refers to the creation of system development versions with minor syntactic code changes. It has high efficiency on defects detecting and it is very flexible in its application and it is being used in various types of artifacts. This work also proposes specific mutation operators for oracles, implements an useful support tool for using these oracle mutation operators and conducts an empirical study of operators, highlighting benefits and challenges associated with their use.

A atividade de teste é essencial para a garantia de qualidade do software e deveria ser empregada durante todo o processo de desenvolvimento. Entretanto, o esforço para a sua aplicação e o alto custo envolvido, comprometem sua utilização de maneira adequada. Durante o processo de desenvolvimento ágil, onde o tempo é um fator crítico, otimizar a atividade de testes sem afetar a qualidade é uma tarefa desafiadora. Apesar do crescente interesse em pesquisas sobre testes no contexto de métodos ágeis, poucas evidências são encontradas sobre avaliação do esforço para elaboração, evolução e manutenção dos testes nesse contexto. Este trabalho propõe uma abordagem para predição de defeitos desenvolvida para o contexto do desenvolvimento ágil e, portanto, considerando as características deste processo de desenvolvimento. Essa abordagem pode ser aplicada quando se considera ou não o desenvolvimento dirigido a testes. A abordagem permite priorizar a execução dos testes com base em uma lista de arquivos que apresentam maior probabilidade de apresentarem defeitos. A abordagem proposta foi avaliada por meio de um estudo de caso conduzido em um ambiente real de desenvolvimento. Como resultado obtido, observou-se que a abordagem melhorou a qualidade do projeto desenvolvido, sem aumentar o esforço durante a atividade de teste de software. / The testing activity is essential to ensure the software quality and should be applied during all steps of the developing process. However the required effort to do this and the high cost involved, compromises its proper usage. During the agile development process, in which time is a critical factor, optimizing the testing activity without affecting quality is a challenge. Despite the growing interest in research based agile method testing, few works are found on the evaluation of the effort to prepare, develop and maintain test cases in this context. This paper proposes an approach for fault prediction in the context of agile development and therefore considering the characteristics of this development process. This approach can be applied both when considering test-driven development or not. It allows prioritizing the execution of tests based on a list containing files most likely to fail. The proposed approach was evaluated by a case study conducted in a real development environment. The results indicate that the approach can improve the quality of the projects without increasing the effort during the testing activity.

Software testing is one of the most important Software Engineering processes, being the primary activity to check the conformance between the software requirements and its actual behavior. The automation of software testing activities is essential to certify productivity and effectiveness in such activities. Test automation leads testing activities to be conducted under systematic and accurate criteria, raising the chance of testers to reveal faults or inconsistencies. Test oracles are elementary members in software testing automation, being the mechanism responsible for indicating the correctness of software outputs. In testing environments, test oracles can be effectively implemented based on several sources of information about the Software Under Testing (SUT): software specifications, assertions, formal methods (Finite State Machines (FSM), formal specifications, etc, machine-learning methods, and metamorphic relations. Regardless of the implementation strategy, test oracles are vulnerable to false positive/negative verdicts, configuring what the literature describes as the oracle problem. Therefore, test oracles are a non-trivial and challenging object of studies of the software engineering research area. SUTs outputs in unusual formats make it harder the oracle problem. Audio, images, three-dimensional objects, virtual reality environments, complex statistical compositions, etc, are examples of non-trivial output formats. In the software testing context, SUTs with unusual outputs can be called complex-output systems. In this doctorate dissertation, we propose and evaluate a novel test oracle approach for complex-output systems called feature-based test oracles. The purpose of feature-based test oracles is the appropriation of a processing image technique called Content-Based Image Retrieval (CBIR) to collect information from features extracted from the SUTs outputs to compose test oracles. Given a query image, CBIR combines feature extraction and similarity functions to alleviate the problem of searching for digital images in large databases. In previous research, we have integrated CBIR concepts in a testing framework to support the automation of testing activities in processing image systems and systems with Graphical User Interfaces (GUI). In this doctorate dissertation, we extended that framework and its concepts to general complex-output systems, addressing the feature-based test oracle approach. We use Text-To-Speech (TTS) systems to validate empirically our test oracle technique. Through the results of five empirical analyses, three of them conducted in line with problems of a real-world industry TTS system, show the proposed technique is a valuable instrument to automate testing activities and alleviate practitioners efforts on testing complex output systems. We conclude the proposed test oracles are effective because they systematically evaluate the SUTs sensorial output rather than produce verdicts based on subjective specifications. As future work, we plan to conduct investigations towards the reduction of false positives/negatives and the association of the test oracles with machine learning techniques and metamorphic relations. / Teste de Software é um dos processos mais importantes da Engenharia de Software, sendo a principal atividade para averiguar a conformidade de requisitos de software e suas saídas. A automatização das atividades de teste é essencial para conferir produtividade e efetividade em tais atividades. A automatização faz com que atividades de teste sejam conduzidas sob critérios sistemáticos e precisos, aumentando a chance dos testadores de revelarem falhas ou inconcistências. Oráculos de teste são membros elementares na automatização do teste de software, sendo o mecanismo responsável por indicar a corretude das saídas do softwre. Em ambientes de teste, oráculos de teste podem ser efetivamente implementados com base em diversos fontes de informação sobre o sistema em teste: especificações de software, assertivas, métodos formais (máquinas de estados finitas, especificações formais, etc), métodos de aprendizagem de máquina e relações metamórficas. Independente da estratégia de implementação, oráculos de teste são vulneráveis a veridictos de falsos positivos/negativos, configurando o que é apresentado na literatura como O problema do Oráculo. Então, na área de engenharia de software, oráculos de teste são objetos de estudo não-triviais e desafiadores. O problema de oráculo é potencializado quando as saídas do sistema em teste são dadas em formatos não triviais como, por exemplo, audio, imagens, objetos tridimensionais, ambientes de realidade virtual, composições estatísticas complexas, etc. No contexto do teste de software, sistemas com saídas não triviais podem ser chamados de sistemas com saídas complexas. Esta tese de doutorado propões e avalia uma nova estratégia de oráculo de teste para sistemas com saídas complexas. O propósito de tal estratégia é a apropriação da técnica de processamento de imagem conhecida como CBIR (Recuperação de Imagem Basead em Conteúdo CBIR) para coletar informações de características extratídas do sistema em teste, compondo oráculos de teste. A partir de uma imagem de busca, o CBIR combina extração de características e funções de similaridade para aliviar problemas de busca em grandes based de imagens digitais. Em pesquisas anteriores, conceitos de CBIR foram integrados em um arcabouço de teste para apoiar a automatização de atividades de teste em systemas de processamento de imagens e sistemas com interfaces gráficas. Esta tese de doutorado estende o arcabouço e seus conceitos para sistemas com saídas complexas em geral. Sistemas Texto-Fala (TTS) foram utlizados para validações empíricas. Os resultados de seis análises empíricas, duas delas condizidas em consonância com problemas de um TTS industrial, revelam que a técnica proposta é um valioso instrumento para automatizar atividaes de teste e aliviar esforços de profissionais da indústria ao teste sistemas com saídas complexas. Conclui-se que a efetividade dos oráculos de teste propostos são devido às sistemáticas análises do conteúdo das saídas dos sistemas em teste, em vez da análises de especificações subjetivas. Os trabalhos futuros vislumbrados devem ser conduzidos no intuito de reduzir número de falsos positivos/negativos e a associação dos oráculos de teste com técnicas de aprendizado de máquina e relações metamórficas.