ITC/USA 2011 Conference Proceedings / The Forty-Seventh Annual International Telemetering Conference and Technical Exhibition / October 24-27, 2011 / Bally's Las Vegas, Las Vegas, Nevada / An automated missile testing environment, reliant on telemetry data, demands automated control of telemetry devices. Software reuse across many missile products (Wikipedia)i and different lab environments requires a software control product that has a simple interface and an ease of modification across different telemetry device vendors. This paper describes a software application that integrates telemetry control/status into automated test and provides a simplified GUI to expedite manual testing. Results from this application show telemetry overhead time reduced by 74%, with a rapid payback on our investment of less than six months.

Since the early days of software testing, automatic test generation has been suggested as a way of allowing tests to be created at a lower cost. However, industrially useful and applicable tools for automatic test generation are still scarce. As a consequence, the evidence regarding the applicability or feasibility of automatic test generation in industrial practice is limited. This is especially problematic if we consider the use of automatic test generation for industrial safety-critical control systems, such as are found in power plants, airplanes, or trains. In this thesis, we improve the current state of automatic test generation by developing a technique based on model-checking that works with IEC 61131-3 industrial control software. We show how automatic test generation for IEC 61131-3 programs, containing both functional and timing information, can be solved as a model checking problem for both code and mutation coverage criteria.  The developed technique has been implemented in the CompleteTest tool. To evaluate the potential application of our technique, we present several studies where the tool is applied to industrial control software. Results show that CompleteTest is viable for use in industrial practice; it is efficient in terms of the time required to generate tests that satisfy both code and mutation coverage and scales well for most of the industrial programs considered. However, our results also show that there are still challenges associated with the use of automatic test generation. In particular, we found that while automatically generated tests, based on code coverage, can exercise the logic of the software as well as tests written manually, and can do so in a fraction of the time, they do not show better fault detection compared to manually created tests. Specifically, it seems that manually created tests are able to detect more faults of certain types (i.e, logical replacement, negation insertion and timer replacement) than automatically generated tests. To tackle this issue, we propose an approach for improving fault detection by using mutation coverage as a test criterion. We implemented this approach in the CompleteTest tool and used it to evaluate automatic test generation based on mutation testing. While the resulting tests were more effective than automatic tests generated based on code coverage, in terms of fault detection, they still were not better than manually created tests. In summary, our results highlight the need for improving the goals used by automatic test generation tools. Specifically, fault detection scores could be increased by considering some new mutation operators as well as higher-order mutations. Our thesis suggests that automatically generated test suites are significantly less costly in terms of testing time than manually created test suites. One conclusion, strongly supported by the results of this thesis, is that automatic test generation is efficient but currently not quite as effective as manual testing. This is a significant progress that needs to be further studied; we need to consider the implications and the extent to which automatic test generation can be used in the development of reliable safety-critical systems.

No description available.

Context: Software testing is expensive, labor intensive and consumes lot of time in a software development life cycle. There was always a need in software testing to decrease the testing time. This also resulted to focus on Automated Software Testing (AST), because using automated testing, with specific tools, this effort can be dramatically reduced and the costs related with testing can decrease [11]. Manual Testing (MT) requires lot of effort and hard work, if we measure in terms of person per month [11]. Automated Software testing helps to decrease the work load by giving some testing tasks to the computers. Computer systems are cheap, they are faster and don‘t get bored and can work continuously in the weekends. Due to this advantage many researches are working towards the Automation of software testing, which can help to complete the task in less testing time [10]. Objectives: The main aims of this thesis is to 1.) To systematically classify contributions within AST. 2.) To identify the different benefits and challenges of AST. 3.) To identify the whether the reported benefits and challenges found in the literature are prevalent in industry. Methods: To fulfill our aims and objectives, we used Systematic mapping research methodology to systematically classify contributions within AST. We also used SLR to identify the different benefits and challenges of AST. Finally, we performed web based survey to validate the finding of SLR. Results: After performing Systematic mapping, the main aspects within AST include purpose of automation, levels of testing, Technology used, different types of research types used and frequency of AST studies over the time. From Systematic literature review, we found the benefits and challenges of AST. The benefits of AST include higher product quality, less testing time, reliability, increase in confidence, reusability, less human effort, reduction of cost and increase in fault detection. The challenges include failure to achieve expected goals, difficulty in maintenance of test automation, Test automation needs more time to mature, false expectations and lack of skilled people for test automation tools. From web survey, it is observed that almost all the benefits and challenges are prevalent in industry. The benefits such as fault detection and confidence are in contrary to the results of SLR. The challenge about the appropriate test automation strategy has 24 % disagreement from the respondents and 30% uncertainty. The reason is that the automation strategy is totally dependent on the test manager of the project. When asked “Does automated software testing fully replace manual testing”, 80% disagree with this challenge. Conclusion: The classification of the AST studies using systematic mapping gives an overview of the work done in the area of AST and also helps to find research coverage in the area of AST. These results can be used by researchers to use the gaps found in the mapping studies to carry on future work. The results of SLR and web survey clearly show that the practitioners clearly realize the benefits and challenges of AST reported in the literature. / Mobile no: +46723069909

Software testing is crucial for uncovering software defects and ensuring software reliability. Symbolic execution has been utilized for automatic test generation to improve testing effectiveness. However, existing test generation techniques based on symbolic execution fail to take full advantage of programs’ rich amount of documentation specifying their input constraints, which can further enhance the effectiveness of test generation. In this paper we present a general approach, Document-Assisted Symbolic Execution (DASE), to improve automated test generation and bug detection. DASE leverages natural language processing techniques and heuristics to analyze programs’ readily available documentation and extract input constraints. The input constraints are then used as pruning criteria; inputs far from being valid are trimmed off. In this way, DASE guides symbolic execution to focus on those inputs that are semantically more important. We evaluated DASE on 88 programs from 5 mature real-world software suites: GNU Coreutils, GNU findutils, GNU grep, GNU Binutils, and elftoolchain. Compared to symbolic execution without input constraints, DASE increases line coverage, branch coverage, and call coverage by 5.27–22.10%, 5.83–21.25% and 2.81–21.43% respectively. In addition, DASE detected 13 previously unknown bugs, 6 of which have already been confirmed by the developers.

A utilização desta abordagem no ciclo de desenvolvimento de uma aplicação web traz algumas vantagens, como ser necessário gerar manualmente apenas o modelo de comportamento de cada funcionalidade da aplicação, (pois os demais artefatos são gerados automaticamente), consumindo menos tempo e estando menos sujeitos a erros, além de prevenir diferentes interpretações dos requisitos pelos stakeholders, desenvolvedores e testadores. O tempo despendido na especificação dos modelos é compensado pelo tempo economizado com a geração dos cenários e do código de testes. / The testing activity throughout software development is fundamental to the pursuit of software quality and reliability, finding faults to be removed. However, despite its importance, software testing is often an underutilized phase in software development. Moreover, tests are proved to be expensive, difficult and problematic when not done in the appropriate way. A new paradigm for software testing is model-driven testing (MDT), which can be defined as software testing where test cases are derived from a model that describes some aspects of the system being tested, such as behavior, for example. This description, often using UML diagrams and/or its profiles, can be processed to produce a set of test cases. Software specifications based on usage scenarios expressed by appropriate UML diagrams are considered significant and effective, because they describe the system’s requirements from an intuitive and visual perspective. Thus, they can be used for the description of acceptance tests, which validate that the system meets user requirements. These specifications also facilitate the automation of this kind of test. Test automation can decrease time spent on testing, thereby reducing the cost of this activity. Thus, this work proposes an approach for automated generation of acceptance tests from U2TP (the UML 2.0 test profile) diagrams for web applications, based on behavior driven development (BDD) paradigm, obtaining acceptance scenarios and executable test code supported by an acceptance testing automation framework. This approach was applied on an actual development environment, by means of an experiment. Using this approach in an web application development cycle has some advantages, such as being required only to manually generate the model of behavior of each application functionality (because other artifacts are generated automatically), thus being less time consuming and less prone to errors, and preventing different interpretations of requirements by stakeholders, developers and testers. The time spent at the models’ specification is compensated by the time saved with the generation of scenarios and test code.

A utilização desta abordagem no ciclo de desenvolvimento de uma aplicação web traz algumas vantagens, como ser necessário gerar manualmente apenas o modelo de comportamento de cada funcionalidade da aplicação, (pois os demais artefatos são gerados automaticamente), consumindo menos tempo e estando menos sujeitos a erros, além de prevenir diferentes interpretações dos requisitos pelos stakeholders, desenvolvedores e testadores. O tempo despendido na especificação dos modelos é compensado pelo tempo economizado com a geração dos cenários e do código de testes. / The testing activity throughout software development is fundamental to the pursuit of software quality and reliability, finding faults to be removed. However, despite its importance, software testing is often an underutilized phase in software development. Moreover, tests are proved to be expensive, difficult and problematic when not done in the appropriate way. A new paradigm for software testing is model-driven testing (MDT), which can be defined as software testing where test cases are derived from a model that describes some aspects of the system being tested, such as behavior, for example. This description, often using UML diagrams and/or its profiles, can be processed to produce a set of test cases. Software specifications based on usage scenarios expressed by appropriate UML diagrams are considered significant and effective, because they describe the system’s requirements from an intuitive and visual perspective. Thus, they can be used for the description of acceptance tests, which validate that the system meets user requirements. These specifications also facilitate the automation of this kind of test. Test automation can decrease time spent on testing, thereby reducing the cost of this activity. Thus, this work proposes an approach for automated generation of acceptance tests from U2TP (the UML 2.0 test profile) diagrams for web applications, based on behavior driven development (BDD) paradigm, obtaining acceptance scenarios and executable test code supported by an acceptance testing automation framework. This approach was applied on an actual development environment, by means of an experiment. Using this approach in an web application development cycle has some advantages, such as being required only to manually generate the model of behavior of each application functionality (because other artifacts are generated automatically), thus being less time consuming and less prone to errors, and preventing different interpretations of requirements by stakeholders, developers and testers. The time spent at the models’ specification is compensated by the time saved with the generation of scenarios and test code.

A utilização desta abordagem no ciclo de desenvolvimento de uma aplicação web traz algumas vantagens, como ser necessário gerar manualmente apenas o modelo de comportamento de cada funcionalidade da aplicação, (pois os demais artefatos são gerados automaticamente), consumindo menos tempo e estando menos sujeitos a erros, além de prevenir diferentes interpretações dos requisitos pelos stakeholders, desenvolvedores e testadores. O tempo despendido na especificação dos modelos é compensado pelo tempo economizado com a geração dos cenários e do código de testes. / The testing activity throughout software development is fundamental to the pursuit of software quality and reliability, finding faults to be removed. However, despite its importance, software testing is often an underutilized phase in software development. Moreover, tests are proved to be expensive, difficult and problematic when not done in the appropriate way. A new paradigm for software testing is model-driven testing (MDT), which can be defined as software testing where test cases are derived from a model that describes some aspects of the system being tested, such as behavior, for example. This description, often using UML diagrams and/or its profiles, can be processed to produce a set of test cases. Software specifications based on usage scenarios expressed by appropriate UML diagrams are considered significant and effective, because they describe the system’s requirements from an intuitive and visual perspective. Thus, they can be used for the description of acceptance tests, which validate that the system meets user requirements. These specifications also facilitate the automation of this kind of test. Test automation can decrease time spent on testing, thereby reducing the cost of this activity. Thus, this work proposes an approach for automated generation of acceptance tests from U2TP (the UML 2.0 test profile) diagrams for web applications, based on behavior driven development (BDD) paradigm, obtaining acceptance scenarios and executable test code supported by an acceptance testing automation framework. This approach was applied on an actual development environment, by means of an experiment. Using this approach in an web application development cycle has some advantages, such as being required only to manually generate the model of behavior of each application functionality (because other artifacts are generated automatically), thus being less time consuming and less prone to errors, and preventing different interpretations of requirements by stakeholders, developers and testers. The time spent at the models’ specification is compensated by the time saved with the generation of scenarios and test code.

Automated Test Data Generation (ATDG) is an activity that in the course of software testing automatically generates test data for the software under test (SUT). It usually makes the testing more efficient and cost effective. Test Data Generation (TDG) is crucial for software testing because test data is one of the key factors for determining the quality of any software test during its execution. The multi-phased activity of ATDG involves various techniques for each of its phases. This research field is not new by any means, albeit lately new techniques have been devised and a gradual increase in the level of maturity has brought some diversified trends into it. To this end several ATDG techniques are available, but emerging trends in computing have raised the necessity to summarize and assess the current status of this area particularly for practitioners, future researchers and students. Further, analysis of the ATDG techniques becomes even more important when Miller et al. [4] highlight the hardship in general acceptance of these techniques. Under this scenario only a systematic review can address the issues because systematic reviews provide evaluation and interpretation of all available research relevant to a particular research question, topic area, or phenomenon of interest. This thesis, by using a trustworthy, rigorous, and auditable methodology, provides a systematic review that is aimed at presenting a fair evaluation of research concerning ATDG techniques of the period 1997-2006. Moreover it also aims at identifying probable gaps in research about ATDG techniques of defined period so as to suggest the scope for further research. This systematic review is basically presented on the pattern of [5 and 8] and follows the techniques suggested by [1].The articles published in journals and conference proceedings during the defined period are of concern in this review. The motive behind this selection is quite logical in the sense that the techniques that are discussed in literature of this period might reflect their suitability for the prevailing software environment of today and are believed to fulfill the needs of foreseeable future. Furthermore only automated and/or semiautomated ATDG techniques have been chosen for consideration while leaving the manual techniques as they are out of the scope. As a result of the preliminary study the review identifies ATDG techniques and relevant articles of the defined period whereas the detailed study evaluates and interprets all available research relevant to ATDG techniques. For interpretation and elaboration of the discovered ATDG techniques a novel approach called ‘Natural Clustering’ is introduced. To accomplish the task of systematic review a comprehensive research method has been developed. Then on the practical implications of this research method important results have been gained. These results have been presented in statistical/numeric, diagrammatic, and descriptive forms. Additionally the thesis also introduces various criterions for classification of the discovered ATDG techniques and presents a comprehensive analysis of the results of these techniques. Some interesting facts have also been highlighted during the course of discussion. Finally, the discussion culminates with inferences and recommendations which emanate from this analysis. As the research work produced in the thesis is based on a rich amount of trustworthy information, therefore, it could also serve the purpose of being an upto- date guide about ATDG techniques. / Shahid Mahmood Folkparksvägen 14:23 372 40 Ronneby Sweden +46 76 2971676

Automating test processes is becoming more and more up-to-date and one of the reasons is that manual tests are time-consuming and complicated. The aim of the study is to identify the manual test process in integration systems to see if there can be a value in automating them and which requirements can then form the basis of an automation. Pros and cons of both manual and automated tests are highlighted in the study in order to make a fair comparison between them. The advantages of the automation that has been highlighted have since, together with collected empirical data at the company Pulsen Integration, been used to produce a requirement list for automation. The study addresses whether it is worth automating all processes or whether some of the test activities work better as manual. By looking at Pulsen Integration's testing processes, one could see that some flexible processes are better as manual, while more general activities can be more easily automated. The study has had a qualitative approach and the chosen respondents from Pulsen Integration contributed with information to be able to produce a relevant list of requirements. We believe that the result presented can be generalized to companies in similar context.