Programs for Programmable Logic Controller (PLC) are often written in graphical or textual languages. Control engineers design and use them in systems where safety is vital, such as avionics, nuclear power plants or transportation systems. Malfunction of such a computer could have severe consequences, therefore thorough testing of PLCs are important. The Base Choice (BC) combination strategy was proposed as a suitable technique for testing software. Test cases are created based on BC strategy by varying the values of one parameter at a time while keeping the values of the other parameters fixed on the values in the base choice. However, this strategy might not be as effective when used on embedded software where parameters need to be set for a certain amount of time in order to trigger a certain interesting behavior. By incorporating time as another parameter when generating the tests, the goal is to create a better strategy that will increase not only code coverage but also fault detection compared to base choice strategy. Timed Base Choice (TBC) coverage criteria is an improvement upon the regular Base Choice criteria with the inclusion of time. We define TBC as follows: The base test case in timed base choice criteria is determined by the tester of the program. A criterion suggested by Ammann and Offutt is the “most likely value” from the point of view of the user. In addition, a time choice T is determined by the tester as the most likely time for keeping the base test case to the same values. From the base test case, new test cases are created by varying the interesting values of one parameter at a time, keeping the values of the other parameters fixed on the base test case. Each new test case is executed with the input values set for a certain amount of time determined by the time choice T. The time choice is given in time units. The research questions stated in this thesis are as follows: Research Question 1 (RQ1) How does Timed Base Choice tests compare to Base Choice tests in terms of decision coverage? Research Question 2 (RQ2) How does Timed Base Choice tests compare to Base Choice tests in terms of fault detection? In order to answer these questions, an empirical study was made in which 11 programs was tested along with respective test cases generated by BC and TBC. Each program was executed on a PLC along with the belonging test cases and several faulty programs (mutants). From this testing we got the corresponding decision coverage for each program achieved by BC and TBC respectively as well as a mutation score measuring how many of the mutated programs was detected and killed. We found that TBC outperformed BC testing both in terms of decision coverage and fault detection. Using TBC testing we managed to achieve full decision coverage on several programs that we were unable to achieve using regular BC. This shows that TBC is an improvement upon the regular BC in both ways, thus answering our previously stated research questions.

Test coverage criteria introduces metrics to measure the adequacy of a test suite as well as defined rules for performing software testing. This makes it possibly to formally define requirements for testing in various industries where software quality is essential, such as in aviation. This bachelor thesis aims to study the effectiveness of 11 different coverage criteria in two ways. Firstly how effective they are at finding faults and secondly their cost effectiveness. Test cases were created for each individual criteria for three different programs. The effectiveness of these test cases was then measured using mutation analysis. The results revealed that the scale of the experiment was too small to truly draw any conclusions regarding effectiveness. It was however shown that due to the “test noise” effect, the effectiveness of the test criteria can differ notably. It was also shown that test coverage criteria alone may not be sufficient in order to perform efficient testing.

Software testing is important, but judging whether a set of software tests is effective is difficult. This problem also appears in the classroom as educators more frequently include software testing activities in programming assignments. The most common measures used to assess student-written software tests are coverage criteria—tracking how much of the student’s code (in terms of statements, or branches) is exercised by the corresponding tests. However, coverage criteria have limitations and sometimes overestimate the true quality of the tests. This dissertation investigates alternative measures of test quality based on how many defects the tests can detect either from code written by other students—all-pairs execution—or from artificially injected changes—mutation analysis. We also investigate a new potential measure called checked code coverage that calculates coverage from the dynamic backward slices of test oracles, i.e. all statements that contribute to the checked result of any test. Adoption of these alternative approaches in automated classroom grading systems require overcoming a number of technical challenges. This research addresses these challenges and experimentally compares different methods in terms of how well they predict defect-detection capabilities of student-written tests when run against over 36,500 known, authentic, human-written errors. For data collection, we use CS2 assignments and evaluate students’ tests with 10 different measures—all-pairs execution, mutation testing with four different sets of mutation operators, checked code coverage, and four coverage criteria. Experimental results encompassing 1,971,073 test runs show that all-pairs execution is the most accurate predictor of the underlying defect-detection capability of a test suite. The second best predictor is mutation analysis with the statement deletion operator. Further, no strong correlation was found between defect-detection capability and coverage measures. / Ph. D.

Submitted by Filipe dos Santos (fsantos@pucsp.br) on 2017-02-22T11:44:10Z No. of bitstreams: 1 Janaina Sarah Pedrotti.pdf: 9898633 bytes, checksum: 6b87849833a694a4591bb4197df9de9f (MD5) / Made available in DSpace on 2017-02-22T11:44:10Z (GMT). No. of bitstreams: 1 Janaina Sarah Pedrotti.pdf: 9898633 bytes, checksum: 6b87849833a694a4591bb4197df9de9f (MD5) Previous issue date: 2017-02-13 / Conselho Nacional de Pesquisa e Desenvolvimento Científico e Tecnológico - CNPq / Fundação São Paulo - FUNDASP / To think about journalism as a process, you have to look at the complex and diffuse paths of its practice to broaden the theoretical understanding of the scientific field. This thesis discusses the theme through the bias of the practice of commercial press with its organizational and structural characteristics and the singular relations. The corpus focuses on the entanglement of the journalistic practice around the press coverage in the socio-environmental area. The process of, production and edition is explored, considering also that these processes have impregnated the collective production dynamics, while it develops in apparent and contradictory individual activity. The adopted perspective is inserted in the discussion about the creation of communication networks processes in different entrances, in order to consider this entanglement of the process of creation in journalism, considering the procedural studies of Cecilia Salles. Texture that led to the expansion of the conception of the journalism making that this thesis brings, by looking at the coverage dynamics of the Folha de S. Paulo, Estado de S. Paulo, Folha do Estado (MT) newspapers and their respective websites with emphasis on the monitoring of the withdraw process in Marãiwetesédé in 2012 and 2013. And later in the journalistic coverage of the rupture of the tailings dam of the mining company Samarco in Bento Rodrigues, district of Mariana (MG) in 2015. The theoretical body is formed by a multidisciplinary framework of two interconnected central axes. The first is supported by a theoretical-methodological approach based on semiotics, subsidized in the formulations of C. Peirce, the complexus of E. Morin and the procedural perspective of C. A. Sales, which allowed us to understand nuances and questions that were not so comprehensible to other approaches. The second axis focuses on the discussions in the field of journalism under guardianship of theoretical such as M. Chaparro, R. Alsina and Medina in conceptual dialogues with the field. This background provided support for the meta-theoretical articulation around the connections in journalistic coverage brought by the dimensions: Expansion of journalistic making; shifts in journalistic coverage - from local to global; mobility of the team; sources of information and conflict and catastrophe in catharsis / Pensar jornalismo enquanto processo é olhar pelos percursos complexos e difusos de sua prática para, desta forma, ampliar a compreensão teórica do campo. A tese discute o tema pelo viés do trabalho da imprensa comercial com suas características organizacionais, estruturais e as relações singulares oriundas da rede acionada pelos sujeitos envoltos nessas dinâmicas. O corpus centra-se no enredamento do fazer jornalístico, em torno da cobertura da imprensa, na área sócio ambiental, explorando seu processo de elaboração, produção e edição. Também considera que estes processos têm impregnado, no seu amago, a dinâmica da produção coletiva, embora se desenvolva em aparente e contraditória atividade individual. A perspectiva adotada insere-se na discussão sobre processos de criação como redes comunicativas em diferentes entradas, de forma a considerar esse enredamento da dinâmica de criação no jornalismo, à luz dos estudos processuais de Cecília Salles. Tessitura que levou à concepção da interação do fazer jornalístico que a tese traz pelo olhar nas dinâmicas de cobertura dos jornais Folha de S. Paulo, O Estado de S. Paulo, Folha do Estado (MT), e seus respectivos sites, com ênfase no acompanhamento do processo de desintrusão em Marãiwatsédé (MT), em 2012 e 2013 e, posteriormente, na cobertura jornalística do rompimento da barragem de rejeitos da mineradora Samarco em Bento Rodrigues, distrito de Mariana (MG), em 2015. O corpo teórico é formado por um quadro multidisciplinar de dois eixos centrais interligados. O primeiro é sustentado por uma abordagem teórico-metodológica, pautada na semiótica, subsidiada pelas formulações de C. S. Peirce, pelo complexus de E. Morin e pela perspectiva processual de C. A. Sales, o que permitiu compreender nuances e questões antes não tão perceptíveis a outras abordagens. O segundo eixo recorre às discussões no campo do jornalismo, acionando-se teóricos como M. Chaparro, R. Alsina e Medina em diálogos conceituais com jornalistas de campo que são referência na atividade. Esse pano de fundo ofereceu sustentação para a articulação meta-teórica em torno das conexões na cobertura jornalística trazidas pelas dimensões: Interação do fazer jornalístico; Deslocamentos na cobertura jornalística- do local ao global; Mobilidade da equipe; Fontes de informação e Conflito e catástrofe em catarse

Este trabalho propõe um modelo de teste para programas concorrentes que utilizam memória compartilhada. O modelo é inovador em três aspectos principais: (1) tratar a sincronização e a comunicação de threads de forma separada, (2) considerar a sincronização decorrente da inicialização/finalização de threads, e (3) apresenta um método baseado em timestamps para determinar as comunicações exercitadas em uma dada execução do programa. Os critérios de cobertura existentes para programas concorrentes foram adaptados ao contexto de programas baseados no paradigma de memória compartilhada. A ferramenta chamada ValiPThread foi implementada neste trabalho para apoiar a aplicação do modelo e dos critérios definidos. Com essa ferramenta é possível criar sessões de teste que podem ser salvas, interrompidas e retomadas a qualquer momento. Também é possível adicionar e executar casos de teste, avaliando a cobertura do código fonte em relação aos critérios de teste. A implementação da ferramenta mostra que é possível instanciar o modelo proposto em um software que auxilie a atividade de teste no contexto de programas com memória compartilhada. O trabalho apresenta soluções significativas para os principais desafios impostos pela programação concorrente para a atividade de teste, destacando-se dentre eles: (1) desenvolvimento de novas técnicas de análise estática para analisar programas concorrentes no contexto de memória compartilhada; (2) testar aspectos cruciais à programação concorrente como: sincronização, comunicação e fluxo de dados; (3) reproduzir uma execução de maneira controlada; (4) mapeamento de critérios de teste já existentes para programas concorrentes com passagem de mensagem para o contexto de memória compartilhada; (5) projetar critérios de fluxo de dados para programas concorrentes, considerando variáveis compartilhadas e (6) desenvolvimento de uma ferramenta de apoio a essas atividades / This work presents a novel test model for shared memory concurrent programs. Some important new features in this model are: (1) analysis the communication and synchronization in an isolated manner, (2) examines the synchronization due the start and the finish of threads, and (3) employs a method based on timestamps to check the communication exercised for an execution of the program. The coverage criteria set defined for concurrent programs was adjusted to the shared memory programs. In this work, the tool ValiPThread was implemented to support the application of the test model and of the coverage criteria. This tool allows to create a test session that can be saved, stopped and resumed at any time. In this tool is also possible to add and to execute test cases, analyzing the source code coverage with respect to the coverage criteria. The new tool shows that is possible instance the proposed model in a software that supports the test activity in context of shared memory. This work presents solutions for the major challenges related to task of to test concurrent programs, such as: (1) develop new techniques to do static analyzes of shared memory programs; (2) test important aspects to concurrent programs such as like: synchronization, communication and data flow; (3) replay and deterministic re-execution; (4) adjust coverage criteria from the context of message passing concurrent programs to shared memory context; (5) design data flow criteria based on shared variables and (6) build a tool to support these activities

Testing is the dominant verification technique used in the software industry today. The use of automatic test case execution increases, but the creation of test cases remains manual and thus error prone and expensive. To automate generation and selection of test cases, model-based testing techniques have been suggested. In this thesis two central problems in model-based testing are addressed: the problem of how to formally specify coverage criteria, and the problem of how to generate a test suite from a formal timed system model, such that the test suite satisfies a given coverage criterion. We use model checking techniques to explore the state-space of a model until a set of traces is found that together satisfy the coverage criterion. A key observation is that a coverage criterion can be viewed as consisting of a set of items, which we call coverage items. Each coverage item can be treated as a separate reachability problem. Based on our view of coverage items we define a language, in the form of parameterized observer automata, to formally describe coverage criteria. We show that the language is expressive enough to describe a variety of common coverage criteria described in the literature. Two algorithms for test case generation with observer automata are presented. The first algorithm returns a trace that satisfies all coverage items with a minimum cost. We use this algorithm to generate a test suite with minimal execution time. The second algorithm explores only states that may increase the already found set of coverage items. This algorithm works well together with observer automata. The developed techniques have been implemented in the tool CoVer. The tool has been used in a case study together with Ericsson where a WAP gateway has been tested. The case study shows that the techniques have industrial strength.

Modeliais paremtas testavimas tampa vis populiaresnis. Sudėtingos programinės įrangos kokybės užtikrinimas ir modeliais orientuotas sistemų kūrimas yra pagrindinės šio testavimo būdo išpopuliarėjimo priežastys. Pagrindiniai šio darbo tikslai yra modeliais paremtų testavimo įrankių analizė ir jų tobulinimo galimybės. Darbe naudojama daugiau nei dešimt įvairių įrankių vertinimo kriterijų. Analizuojama testų padengimo kriterijų pritaikymo ir tobulinimo galimybės. Analizuojamos modeliais paremto proceso problemos ir pateikiami sprendimo būdai. Atlikta detali testų kūrimo kriterijų ir modelio perėjimo algoritmų analizė. Atlikus detalę kriterijų ir modelio perėjimo algoritmų analizę, pateikiama kriterijų apjungimo privalumai ir kaip naudojant įvairius kriterijus galima valdyti testų kūrimo procesą ir siekti 100 procentinio modelio padengimo. Pateikiama ir detaliai išanalizuojama kriterijų kūrimo metodika, kurią galim taikyti testavimo įrankiuose. / Model-based testing has become increasingly popular in recent years. Major reasons include the need for quality assurance for increasingly complex systems and the emerging model-centric development paradigm. The present thesis aims at analysing model-based testing tools and presents the possibilities for their improvement. More than dozen criteria are applied in analysing selected tools. The paper analyses the application of testing coverage criteria and presents their possible improvements. The problems occurring in the model-based testing process are being analysed and the solutions presented. The study contains an in-depth analysis of test coverage criteria and model traversal algorithms. A thorough analysis of test criteria and model traversal algorithms being performed, the advantages of the integration of test criteria are presented. Furthermore, a solution for managing the development process of test cases as well as pursuing a hundred percent model coverage is proposed. The paper presents a comprehensive analysis of test criteria development methodology that could be applied in testing tools.

Este trabalho propõe um modelo de teste para programas concorrentes que utilizam memória compartilhada. O modelo é inovador em três aspectos principais: (1) tratar a sincronização e a comunicação de threads de forma separada, (2) considerar a sincronização decorrente da inicialização/finalização de threads, e (3) apresenta um método baseado em timestamps para determinar as comunicações exercitadas em uma dada execução do programa. Os critérios de cobertura existentes para programas concorrentes foram adaptados ao contexto de programas baseados no paradigma de memória compartilhada. A ferramenta chamada ValiPThread foi implementada neste trabalho para apoiar a aplicação do modelo e dos critérios definidos. Com essa ferramenta é possível criar sessões de teste que podem ser salvas, interrompidas e retomadas a qualquer momento. Também é possível adicionar e executar casos de teste, avaliando a cobertura do código fonte em relação aos critérios de teste. A implementação da ferramenta mostra que é possível instanciar o modelo proposto em um software que auxilie a atividade de teste no contexto de programas com memória compartilhada. O trabalho apresenta soluções significativas para os principais desafios impostos pela programação concorrente para a atividade de teste, destacando-se dentre eles: (1) desenvolvimento de novas técnicas de análise estática para analisar programas concorrentes no contexto de memória compartilhada; (2) testar aspectos cruciais à programação concorrente como: sincronização, comunicação e fluxo de dados; (3) reproduzir uma execução de maneira controlada; (4) mapeamento de critérios de teste já existentes para programas concorrentes com passagem de mensagem para o contexto de memória compartilhada; (5) projetar critérios de fluxo de dados para programas concorrentes, considerando variáveis compartilhadas e (6) desenvolvimento de uma ferramenta de apoio a essas atividades / This work presents a novel test model for shared memory concurrent programs. Some important new features in this model are: (1) analysis the communication and synchronization in an isolated manner, (2) examines the synchronization due the start and the finish of threads, and (3) employs a method based on timestamps to check the communication exercised for an execution of the program. The coverage criteria set defined for concurrent programs was adjusted to the shared memory programs. In this work, the tool ValiPThread was implemented to support the application of the test model and of the coverage criteria. This tool allows to create a test session that can be saved, stopped and resumed at any time. In this tool is also possible to add and to execute test cases, analyzing the source code coverage with respect to the coverage criteria. The new tool shows that is possible instance the proposed model in a software that supports the test activity in context of shared memory. This work presents solutions for the major challenges related to task of to test concurrent programs, such as: (1) develop new techniques to do static analyzes of shared memory programs; (2) test important aspects to concurrent programs such as like: synchronization, communication and data flow; (3) replay and deterministic re-execution; (4) adjust coverage criteria from the context of message passing concurrent programs to shared memory context; (5) design data flow criteria based on shared variables and (6) build a tool to support these activities

<p>Testing is the dominant verification technique used in the software industry today. The use of automatic test case execution increases, but the creation of test cases remains manual and thus error prone and expensive. To automate generation and selection of test cases, model-based testing techniques have been suggested.</p><p>In this thesis two central problems in model-based testing are addressed: the problem of how to formally specify coverage criteria, and the problem of how to generate a test suite from a formal timed system model, such that the test suite satisfies a given coverage criterion. We use model checking techniques to explore the state-space of a model until a set of traces is found that together satisfy the coverage criterion. A key observation is that a coverage criterion can be viewed as consisting of a set of items, which we call coverage items. Each coverage item can be treated as a separate reachability problem. </p><p>Based on our view of coverage items we define a language, in the form of parameterized observer automata, to formally describe coverage criteria. We show that the language is expressive enough to describe a variety of common coverage criteria described in the literature. Two algorithms for test case generation with observer automata are presented. The first algorithm returns a trace that satisfies all coverage items with a minimum cost. We use this algorithm to generate a test suite with minimal execution time. The second algorithm explores only states that may increase the already found set of coverage items. This algorithm works well together with observer automata.</p><p>The developed techniques have been implemented in the tool CoVer. The tool has been used in a case study together with Ericsson where a WAP gateway has been tested. The case study shows that the techniques have industrial strength.</p>

Les travaux de cette thèse contribuent au développement de méthodes automatiques de vérification et de valida-tion de systèmes informatiques, à partir de modèles. Ils sont divisés en deux parties : vérification et générationde tests.Dans la partie vérification, pour le problème du model-checking régulier indécidable en général, deux nouvellestechniques d’approximation sont définies, dans le but de fournir des (semi-)algorithmes efficaces. Des sur-approximations de l’ensemble des états accessibles sont calculées, avec l’objectif d’assurer la terminaison del’exploration de l’espace d’états. Les états accessibles (ou des sur-approximations de cet ensemble d’états)sont représentés par des langages réguliers, ou automates d’états finis. La première technique consiste à sur-approximer l’ensemble des états atteignables en fusionnant des états des automates, en fonction de critèressyntaxiques simples, ou d’une combinaison de ces critères. La seconde technique d’approximation consisteaussi à fusionner des états des automates, mais à l’aide de transducteurs. De plus, pour cette seconde technique,nous développons une nouvelle approche pour raffiner les approximations, qui s’inspire du paradigme CEGAR(CounterExample-Guided Abstraction Refinement). Ces propositions ont été expérimentées sur des exemplesde protocoles d’exclusion mutuelle.Dans la partie génération de tests, une technique qui permet de combiner la génération aléatoire avec un critèrede couverture, à partir de modèles algébriques (des grammaires algébriques, des automates à pile) est définie.Générer les tests à partir de ces modèles algébriques (au lieu de le faire à partir de graphes) permet de réduirele degré d’abstraction du modèle et donc de générer moins de tests qui ne sont pas exécutables dans le systèmeréel. Ces propositions ont été expérimentées sur la grammaire de JSON (JAvaScript Object Notation), ainsi quesur des automates à pile correspondant à des appels de fonctions mutuellement récursives, à une requête XPath,et à l’algorithme Shunting-Yard. / The thesis contributes to development of automatic methods for model-based verification and validation ofcomputer systems. It is divided into two parts: verification and test generation.In the verification part, for the problem of regular model checking undecidable in general, two new approxi-mation techniques are defined in order to provide efficient (semi-)algorithms. Over-approximations of the setof reachable states are computed, with the objective of ensuring the termination of the exploration of the statespace. Reachable states (or over-approximations of this set of states) are represented by regular languages or,equivalently, by finite-state automata. The first technique consists in over-approximating the set of reachablestates by merging states of automata, based on simple syntactic criteria, or on a combination of these criteria.The second approximation technique also merges automata states, by using transducers. For the second tech-nique, we develop a new approach to refine approximations, inspired by the CEGAR paradigm (for Counter-Example-Guided Abstraction Refinement). These proposals have been tested on examples of mutual exclusionprotocols.In the test generation part, a technique that combines the random generation with coverage criteria, fromcontext-free models (context-free grammars, pushdown automata) is defined. Generate tests from these mo-dels (instead of doing from graphs) reduces the model abstraction level, and therefore allows having moretests executable in the real system. These proposals have been tested on the JSON grammar (JavaScript ObjectNotation), as well as on pushdown automata of mutually recursive functions, of an XPath query, and of theShunting-Yard algorithm.