As atividades de teste de software são de crescente importância devido à maciça presença de sistemas de informação em nosso cotidiano. Programas de Processamento de Imagens (PI) têm um domínio de entrada bastante complexo e, por essa razão, o teste tradicional realizado com esse tipo de programa, conduzido majoritariamente de forma manual, é uma tarefa de alto custo e sujeita a imperfeições. No teste tradicional, em geral, as imagens de entrada são construídas manualmente pelo testador ou selecionadas aleatoriamente de bases de imagens, muitas vezes dificultando a revelação de defeitos no software. A partir de um mapeamento sistemático da literatura realizado, foi identificada uma lacuna no que se refere à geração automatizada de dados de teste no domínio de imagens. Assim, o objetivo desta pesquisa é propor uma abordagem - denominada TAIGA (Test imAge generatIon by Genetic Algorithm) - para a geração de dados de teste para programas de PI por meio de algoritmo genético. Na abordagem proposta, operadores genéticos tradicionais (mutação e crossover) são adaptados para o domínio de imagens e a função fitness é substituída por uma avaliação de resultados provenientes de teste de mutação. A abordagem TAIGA foi validada por meio de experimentos com oito programas de PI distintos, nos quais observaram-se ganhos de até 38,61% em termos de mutation score em comparação ao teste tradicional. Ao automatizar a geração de dados de teste, espera-se conferir maior qualidade ao desenvolvimento de sistemas de PI e contribuir com a diminuição de custos com as atividades de teste de software neste domínio / The massive presence of information systems in our lives has been increasing the importance of software test activities. Image Processing (IP) programs have very complex input domains and, therefore, the traditional testing for this kind of program is a highly costly and vulnerable to errors task. In traditional testing, usually, testers create images by themselves or they execute random selection from images databases, which can make it harder to reveal faults in the software under test. In this context, a systematic mapping study was conducted and a gap was identified concerning the automated test data generation in the images domain. Thus, an approach for generating test data for IP programs by means of genetic algorithms was proposed: TAIGA - Test imAge generatIon by Genetic Algorithm. This approach adapts traditional genetic operators (mutation and crossover) to the images domain and replaces the fitness function by the evaluation of the results of mutation testing. The proposed approach was validated by the execution of experiments involving eight distinct IP programs. TAIGA was able to provide up to 38.61% increase in mutation score when compared to the traditional testing for IP programs. It\'s expected that the automation of test data generation elevates the quality of image processing systems development and reduces the costs of software test activities in the images domain

As atividades de teste de software são de crescente importância devido à maciça presença de sistemas de informação em nosso cotidiano. Programas de Processamento de Imagens (PI) têm um domínio de entrada bastante complexo e, por essa razão, o teste tradicional realizado com esse tipo de programa, conduzido majoritariamente de forma manual, é uma tarefa de alto custo e sujeita a imperfeições. No teste tradicional, em geral, as imagens de entrada são construídas manualmente pelo testador ou selecionadas aleatoriamente de bases de imagens, muitas vezes dificultando a revelação de defeitos no software. A partir de um mapeamento sistemático da literatura realizado, foi identificada uma lacuna no que se refere à geração automatizada de dados de teste no domínio de imagens. Assim, o objetivo desta pesquisa é propor uma abordagem - denominada TAIGA (Test imAge generatIon by Genetic Algorithm) - para a geração de dados de teste para programas de PI por meio de algoritmo genético. Na abordagem proposta, operadores genéticos tradicionais (mutação e crossover) são adaptados para o domínio de imagens e a função fitness é substituída por uma avaliação de resultados provenientes de teste de mutação. A abordagem TAIGA foi validada por meio de experimentos com oito programas de PI distintos, nos quais observaram-se ganhos de até 38,61% em termos de mutation score em comparação ao teste tradicional. Ao automatizar a geração de dados de teste, espera-se conferir maior qualidade ao desenvolvimento de sistemas de PI e contribuir com a diminuição de custos com as atividades de teste de software neste domínio / The massive presence of information systems in our lives has been increasing the importance of software test activities. Image Processing (IP) programs have very complex input domains and, therefore, the traditional testing for this kind of program is a highly costly and vulnerable to errors task. In traditional testing, usually, testers create images by themselves or they execute random selection from images databases, which can make it harder to reveal faults in the software under test. In this context, a systematic mapping study was conducted and a gap was identified concerning the automated test data generation in the images domain. Thus, an approach for generating test data for IP programs by means of genetic algorithms was proposed: TAIGA - Test imAge generatIon by Genetic Algorithm. This approach adapts traditional genetic operators (mutation and crossover) to the images domain and replaces the fitness function by the evaluation of the results of mutation testing. The proposed approach was validated by the execution of experiments involving eight distinct IP programs. TAIGA was able to provide up to 38.61% increase in mutation score when compared to the traditional testing for IP programs. It\'s expected that the automation of test data generation elevates the quality of image processing systems development and reduces the costs of software test activities in the images domain

Differentielles Testen ist ein wichtiger Bestandteil der Qualitätssicherung von Software, mit dem Ziel Testeingaben zu generieren, die Unterschiede im Verhalten der Software deutlich machen. Solche Unterschiede können zwischen zwei Ausführungspfaden (1) in unterschiedlichen Programmversionen, aber auch (2) im selben Programm auftreten. In dem ersten Fall werden unterschiedliche Programmversionen mit der gleichen Eingabe untersucht, während bei dem zweiten Fall das gleiche Programm mit unterschiedlichen Eingaben analysiert wird. Die Regressionsanalyse, die Side-Channel Analyse, das Maximieren der Ausführungskosten eines Programms und die Robustheitsanalyse von Neuralen Netzwerken sind typische Beispiele für differentielle Softwareanalysen. Eine besondere Herausforderung liegt in der effizienten Analyse von mehreren Programmpfaden (auch über mehrere Programmvarianten hinweg). Die existierenden Ansätze sind dabei meist nicht (spezifisch) dafür konstruiert, unterschiedliches Verhalten präzise hervorzurufen oder sind auf einen Teil des Suchraums limitiert. Diese Arbeit führt das Konzept des hybriden differentiellen Software Testens (HyDiff) ein: eine hybride Analysetechnik für die Generierung von Eingaben zur Erkennung von semantischen Unterschieden in Software. HyDiff besteht aus zwei parallel laufenden Komponenten: (1) einem such-basierten Ansatz, der effizient Eingaben generiert und (2) einer systematischen Analyse, die auch komplexes Programmverhalten erreichen kann. Die such-basierte Komponente verwendet Fuzzing geleitet durch differentielle Heuristiken. Die systematische Analyse basiert auf Dynamic Symbolic Execution, das konkrete Eingaben bei der Analyse integrieren kann. HyDiff wird anhand mehrerer Experimente evaluiert, die in spezifischen Anwendungen im Bereich des differentiellen Testens ausgeführt werden. Die Resultate zeigen eine effektive Generierung von Testeingaben durch HyDiff, wobei es sich signifikant besser als die einzelnen Komponenten verhält. / Differential software testing is important for software quality assurance as it aims to automatically generate test inputs that reveal behavioral differences in software. The concrete analysis procedure depends on the targeted result: differential testing can reveal divergences between two execution paths (1) of different program versions or (2) within the same program. The first analysis type would execute different program versions with the same input, while the second type would execute the same program with different inputs. Therefore, detecting regression bugs in software evolution, analyzing side-channels in programs, maximizing the execution cost of a program over multiple executions, and evaluating the robustness of neural networks are instances of differential software analysis with the goal to generate diverging executions of program paths. The key challenge of differential software testing is to simultaneously reason about multiple program paths, often across program variants, in an efficient way. Existing work in differential testing is often not (specifically) directed to reveal a different behavior or is limited to a subset of the search space. This PhD thesis proposes the concept of Hybrid Differential Software Testing (HyDiff) as a hybrid analysis technique to generate difference revealing inputs. HyDiff consists of two components that operate in a parallel setup: (1) a search-based technique that inexpensively generates inputs and (2) a systematic exploration technique to also exercise deeper program behaviors. HyDiff’s search-based component uses differential fuzzing directed by differential heuristics. HyDiff’s systematic exploration component is based on differential dynamic symbolic execution that allows to incorporate concrete inputs in its analysis. HyDiff is evaluated experimentally with applications specific for differential testing. The results show that HyDiff is effective in all considered categories and outperforms its components in isolation.

Autonomy is the most demanded yet hard-to-achieve feature of recent and future software systems. Self-driving cars, mail-delivering drones, automated guided vehicles in production sites, and housekeeping robots need to decide autonomously during most of their operation time. As soon as human intervention becomes necessary, the cost of ownership increases, and this must be avoided. Although the algorithms controlling autonomous systems become more and more intelligent, their hardest opponent is their inflexibility. The more environmental situations such a system is confronted with, the more complexity the control of the autonomous system will have to master. To cope with this challenge, engineers have approached a system design, which adopts feedback loops from nature. The resulting architectural principle, which they call self-adaptive systems, follows the idea of iteratively gathering sensor data, analyzing it, planning new adaptations of the system, and finally executing the plan. Often, adaptation means to alter the system setup, re-wire components, or even exchange control algorithms to keep meeting goals and requirements in the newly appeared situation. Although self-adaptivity helps engineers to organize the vast amount of information in a self-deciding system, it remains hard to deal with the variety of contexts, which involve both environmental influences and knowledge about the system\'s internals. This challenge not only holds for the construction phase but also for verification and validation, including software test. To assure sufficient quality of a system, it must be tested under an enormous and, thus, unmanageable, number of different contextual situations and manual test-cases. This thesis proposes a novel set of methods and model types, which help test engineers to specify precisely what they expect from a self-adaptive system under test. The formal nature of the introduced artifacts allows for automatically generating test-suites or running simulations in the loop so that a qualitative verdict on the system\'s correctness can be gained. Additional to these conceptional contributions, the thesis describes a model-based adaptivity test environment, which test engineers can use for testing actual self-adaptive systems. The implementation includes comprehensive tooling for creating the introduced types of models, generating test-cases, simulating them in the loop, automating tests, and reporting. Composing all enabling components for these tasks constitutes a reference architecture of integrated test environments for self-adaptive systems. We demonstrate the completeness and accuracy of the technical approach together with the underlying concepts by evaluating them in an experimental case study where an autonomous robot interacts with human co-workers. In summary, this thesis proposes concepts for automatically and, thus, efficiently testing self-adaptive systems. The quality, which is fostered by this novel approach, is resilience: the ability of a system to maintain its promises while facing changing environments.:1 Introduction 1 1.1 Problem Description 1 1.2 Overview of Adopted Methods 3 1.3 Hypothesis and Main Contributions 4 1.4 Organization of This Thesis 5 I Foundations 7 2 Background 9 2.1 Self-adaptive Software and Autonomic Computing 9 2.1.1 Common Principles and Components of SAS 10 2.1.2 Concrete Implementations and Applications of SAS 12 2.2 Model-based Testing 13 2.2.1 Testing for Dependability 14 2.2.2 The Basics of Testing 15 2.2.3 Automated Test Design 18 2.3 Dynamic Variability Management 22 2.3.1 Software Product Lines 23 2.3.2 Dynamic Software Product Lines 25 3 Related Work: Existing Research on Testing Self-Adaptive Systems 29 3.1 Testing Context-Aware Applications 30 3.2 The SimSOTA Project 31 3.3 Dynamic Variability in Complex Adaptive Systems (DiVA) 33 3.4 Other Early-Stage Research 34 3.5 Taxonomy of Requirements of Model-based SAS Testing 36 II Methods 39 4 Model-driven SAS Testing 41 4.1 Problem/Solution Fit 41 4.2 Example: Surveillance Drone 43 4.3 Concepts and Models for Testing Self-Adaptive Systems 44 4.3.1 Test Case Generation vs. Simulation in the Loop 44 4.3.2 Incremental Modeling Process 45 4.3.3 Basic Representation Format: Petri Nets 46 4.3.4 Context Variation 50 4.3.5 Modeling Adaptive Behavior 53 4.3.6 Dynamic Context Change 57 4.3.7 Interfacing Context from Behavioral Representation 62 4.3.8 Adaptation Mode Variation 64 4.3.9 Context-Dependent Recon guration 67 4.4 Adequacy Criteria for SAS Test Models 71 4.5 Discussion on the Viability of the Employed Models 71 4.6 Comparison to Related Work 73 4.7 Summary and Discussion 74 5 Model-based Adaptivity Test Environment 75 5.1 Technological Foundation 76 5.2 MATE Base Components 77 5.3 Metamodel Implementation 78 5.3.1 Feature-based Variability Model 79 5.3.2 Abstract and Concrete Syntax for Textual Notations 80 5.3.3 Adaptive Petri Nets 86 5.3.4 Stimulus and Recon guration Automata 87 5.3.5 Test Suite and Report Model 87 5.4 Test Generation Framework 87 5.5 Test Automation Framework 91 5.6 MATE Tooling and the SAS Test Process 93 5.6.1 Test Modeling 94 5.6.2 Test Case Generation 95 5.6.3 Test Case Execution and Test Reporting 96 5.6.4 Interactive Simulation Frontend 96 5.7 Summary and Discussion 97 III Evaluation 99 6 Experimental Study: Self-Adaptive Co-Working Robots 101 6.1 Robot Teaching and Co-Working with WEIR 103 6.1.1 WEIR Hardware Components 104 6.1.2 WEIR Software Infrastructure 105 6.1.3 KUKA LBR iiwa as WEIR Manipulator 106 6.1.4 Self-Adaptation Capabilities of WEIR 107 6.2 Cinderella as Testable Co-Working Application 109 6.2.1 Cinderella Setup and Basic Functionality 109 6.2.2 Co-Working with Cinderella 110 6.3 Testing Cinderella with MATE 112 6.3.1 Automating Test Execution 112 6.3.2 Modeling Cinderella in MATE 113 6.3.3 Testing Cinderella in the Loop 121 6.4 Evaluation Verdict and Summary 123 7 Summary and Discussion 125 7.1 Summary of Contributions 126 7.2 Open Research Questions 127 Bibliography 129 Appendices 137 Appendix Cinderella De nitions 139 1 Cinderella Adaptation Bounds 139 2 Cinderella Self-adaptive Workflow 140