Because software testing is a repetitive and time-intensive task, a practical solution is to turn to automation. Test automation, however, requires programming skills. Testers, who typically know a lot about the application under test, often do not have the programming skills to automate the testing effort. Grammar-based test generation (GBTG) uses context-free grammars to generate strings in the language described by the grammar. Given a grammar, a GBTG algorithm can produce test cases for the application under test. Since testers typically have little programming skills and are not likely to develop the grammars needed for practical testing, the power of GBTG is unavailable to many testing practitioners. To help address this problem, we have developed Dervish, a graphical user interface which allows testers to use the power of GBTG. Our new tool allows testers to modify parts of a grammar, generate test cases, and visualize generation trees. To demonstrate the benefits of Dervish, we present the results of three case studies.

Testing is a necessary component of software development, but it is also an expensive one, especially if performed manually. One way to mitigate the cost of testing is to implement test automation, where the test cases are run automatically. For any organisation looking to implement test automation, the most interesting cost is time. Automation takes time to implement and one of the most obvious benefits of automation is that the automated test execution time is lower than that of manual execution. This thesis contains a literature study covering testing methodology, especially in regards to the domain of web application testing. The literature covered also included three economic models that may be used to calculate the costs of automation compared to manual testing. The models can be used to calculate the time it would take, or the number of necessary executions, for the total cost of test automation to be lower than of that of manual testing. The thesis is based on a case study of test automation for the StoredSafe platform, a web application. Three sets of test automation frameworks were used to implement three different test suits and the test implementation times were collected. The data collected were then used to calculate the time it would take, using the three economic models, for the cost of automated test cases to become equal to that of with manual testing. The data showed that the estimated time to reach breakeven for the three frameworks varied between 2½ and at worst 10 years, with an average of 3½ years. The models and data presented in this thesis may be used in order to estimate the cost of test automation in comparison to manual testing over longer periods of time, but care must be taken in order to ensure that the data used is correct in regards to one’s own organisation or else the estimate may be faulty.

Today’s technology has a strong tendency towards automation. As a matter of fact, the tremendous improvement of science in the recent years brought new ideas regarding to accelerate the scientific process that is not separated of automation. This paper also deals with automation of manual tests that were used to analyze packet loss and packet loss duration in a network. These two metrics were chosen based on their importance they have in the communication technology, also based on the weak points that were found in the manual processes. This experiment is done in the Ixia platform that was an appropriate test bed to design an automation framework.After a comprehensive research on network and communication we could choose packet loss and packet loss duration as two important parameters that are under test several times per day. Therefore, based on the properties that are used for automation, these two metrics had the priority compare to other metrics. We could create a framework that works correspond to the manual test process. For this purpose, Tcl programming language is used. The script that was written with this high-level language can communicate with the graphical user interface, configuring all the connected devices, measuring mentioned metrics and ultimately save the result in a csv file.Finally, we could reach to the main objective of this project which was to show how positively automatic method can affect on the quality of test in terms of accuracy, time and manpower saving.

Made available in DSpace on 2015-04-22T22:00:48Z (GMT). No. of bitstreams: 1 Eliane Figueiredo Collins.pdf: 2314673 bytes, checksum: 8403bd67cffed913ac9029447401f980 (MD5) Previous issue date: 2013-04-25 / CAPES - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior / For some time, the professional software development companies are looking for new technical alternatives in order to optimize their processes, to deliver products to market as soon as possible and to meet customer expectations, increasingly demanding and intolerant of software failures. In this sense, the agile software development is gaining more space and consequently the software testing area which was previously considered a separate phase of development has changed over time to adapt to this new reality. The test automation activity is seen as key of agile development, but many teams still don t know how it should be done, who should take responsibility and run it and the correct procedures for this activity to be successful. Although the literature mentions about the particularities activities and best practices in this scenario, there are few related works and lack of experience reports or case studies that show, in practice, examples of use this from start to finish the automation strategy adopted. This paper aims to contribute to improving the quality of agile processes, and proposes an approach that involves the application of agile manifesto values in the activities of test automation, which can be used regardless of agile methodology adopted. In this study, observations were made on empirical about testing practices in agile projects developed at Nokia Technology Institute (INdT) and an experimental implementation of this approach at the Center for Electronic and Information Technology (CETELI), Federal University of Amazonas (UFAM). Experimental reports are available with different strategies for the automation of software testing and identification of some important issues to deal with difficulties in adapting testers and developers in this new scenario. / Há algum tempo as empresas desenvolvedoras de software profissional vêm buscando novas alternativas técnicas com o objetivo de otimizar seus processos, entregar produtos para o mercado o mais cedo possível e ainda atender as expectativas dos clientes cada vez mais exigentes e intolerantes a falhas de software. Com isso, as metodologias ágeis de desenvolvimento de software estão ganhando mais espaço e consequentemente a área de testes de software que antes era considerada uma fase separada do desenvolvimento vem passando por mudanças para se adaptar a esta nova realidade. A atividade de automação de teste passou a ser vista como peça chave para o desenvolvimento ágil, porém muitas equipes ainda não sabem como ela deve ser feita, quem deve assumir a responsabilidade e execução dela e os procedimentos para que essa atividade seja bem sucedida. Embora a literatura mencione particularidades sobre as atividades realizadas e as melhores práticas nesse cenário, são poucos os trabalhos relacionados e há falta de relatos de estudos de caso ou exemplos de utilização que mostrem, na prática, do início ao fim, a estratégia de automação adotada. Este trabalho tem como objetivo contribuir para a melhoria da qualidade dos processos ágeis, propondo uma abordagem que envolve a aplicação de valores presentes no manifesto ágil nas atividades de automação de teste, que podem ser utilizadas independentemente da metodologia ágil adotada. Nesta pesquisa, foram realizadas observações empíricas sobre as práticas de testes em projetos ágeis desenvolvidos no Instituto Nokia de Tecnologia (INdT) e um experimento de aplicação dessa abordagem no Centro de Tecnologia Eletrônica e da Informação (CETELI), da Universidade Federal do Amazonas (UFAM). São disponibilizados relatos experimentais com diferentes estratégias relativas a automação de teste de software e a identificação de algumas questões importantes para lidar

The products The Company manufactures are used in an area of the industry where errors leading to a stop in production can be quite expensive. Therefore are testing of the products important and the tests can also give indications about the quality of the products. The Company is in a phase where they are developing a new product line to support all existing and future products. In this phase, it was decided that all products will use a common framework for unit testing and a common build system for all products. One part of the thesis was to investigate and evaluate different frameworks for unit testing and tools for a build system. The unit test framework that were evaluated are Cppunit, cfix, NUnit, Boost test library, Unit Testing++ and CxxTest. The result of the evaluation was that CppUnit were recommended. For the build system MSBuild, NAnt, Automated Build Studio and Cruise Control .NET were evaluated. The recommended tools for a build system is MSBuild in combination with Cruise Control .Net if The Company is interested in the functionality Cruise Control .Net has to offer. The Company also has an interest in evaluating the current test processes and identify improvements as a part of The Company’ s objective were all products should follow a common test process. In order to identify these improvements a literature study of four test process improvement frameworks (Test Process Improvement, Test Maturity Model Integrated, Minimal Test Practice Framework and Test Improvement Model) were carried out. Out of these four frameworks Test Process Improvement (TPI) were chosen to assist when identifying improvements. With the help of TPI a limited assessment took place to give indications about the test maturity for three of The Company’s products where two of the products had low maturity. Results of the improvement measures can be summed up with the need to harmonize the documents, standardize and document the various processes. As a last part of the thesis the possibility to automate testing of two of the products graphical user interfaces with the program Test Complete were investigated. For one of the products the result was that it worked satisfactorily and for the other product it did not work at all. This resulted in recommendations for how The Company should proceed with automated testing of the graphical user interface. / De produkter som företaget tillverkar används i huvudsak inom ett område av branschen där fel som leder till stopp i produktionen kan vara ganska dyrt. Detta gör testning av produkterna viktiga och tester kan också ge indikationer om kvaliteten på produkterna. Företaget är i en fas där man utvecklar en ny produktlinje som ska stödja alla befintliga och framtida produkter. I denna fas har man beslutat att alla produkterna ska använda ett gemensant ramverk för enhetstestning och även använda ett gemensamt byggsystem för samtliga produkter. En del av examensarbetet var att undersöka och utvärdera olika ramverk för enhets testning och verktyg för byggsystem. De ramverk som utvärderades var CppUnit, cfix, NUnit, Boost test library, unitTest++ och CxxTest. Utvärderingen ledde fram till att CppUnit rekommenderades till företaget. Verktyg som utvärderades för byggsystem var MSBuild, NAnt, Automated Build Studio och Cruise Control .Net. För byggsystem rekommenderas MSBuild i kombination med Cruise Control .Net ifall företaget är intresserade av den extra funktionalitet som Cruise Control .Net har att erbjuda. Företaget har även ett intresse av att utvärdera den nuvarande testprocessen och identifiera förbättringar som ett led i att befintliga produkter skall följa en gemensam testprocess. För att kunna identifiera dessa förbättringar utfördes en litteraturstudie över fyra stycken test process förbättrings ramverk (Test Process Improvent, Test Maturity Model integrated, Minimal Test Practice Framework och Test Improvement Model). Utav dessa fyra ramverk så valdes Test Process Improvement (TPI) ut som en hjälp för att identifiera förbättringar. Med hjälp av TPI utfördes en begränsad bedömning av företagets mogenhetsgrad på tre produkter, där två av produkterna har en låg mogenhetsgrad. Resultat av förbättringsåtgärderna kan sammanfattas med att man bör harmonisera dokument, standardisera och dokumentera olika processer. Som en sista del i examensarbetet så utvärderades möjligheten att automatisera testning på två av produkternas grafiska användargränssnitt med programmet TestComplete. För en av produkterna blev resultatet att det fungerade tillfredställande och för den andra produkten så fungerade det inte alls. Resultatet blev rekommendationer för hur företaget borde gå vidare med automatisering av testnig på det grafiska användargränssnittet.

The aim of this Master Thesis Frameworks for automatic testing is to compare frameworks of five most popular programming languages of today. The comparison is divided into two parts. The first one deals with comparing by using the test patterns and the second part deals with measuring of exact values while working with tools. The outcome of this thesis are criteria that can be used for comparison of other frameworks for automatic testing other than used in this work.

This thesis deals with software testing and its possible automation and compares the pros and cons of automated testing with manual testing. In the automated approach is introduced tool for automated testing HP QuickTest Professional (HP QTP). The main objective is to analyze the advantages of using this tool in conditions of a real project, based on predefined criteria. In the theoretical section I introduce basic terms within the field of software testing and identify testing in the software development process. A comparison of manual and automated approach to software testing and introduction of tool HP QTP for automated tests follows. Target of practical part is to analyze the advantages of automation by using HP QTP on a real project according to defined criteria and recommend for what conditions the automation in this case worthwhile. In the practical part is also created manual which describing the basic functionality of HP QTP, which is demonstrated on examples. Manual, which is also the output of this work, has to help testers to quickly understand the work with this tool and therefore better plan process of tests automation.

This master's thesis is aimed at creating a methodology for the selection of suitable test for automation testing. The first part is devoted to the definition of the prerequisites for introduction of automated testing and the conditions for its effective course. The second part focuses on defining the exclusion criteria of tests which are unsuitable for automation. The following third part is devoted to defining the selection criteria for the determination of the test priority for automation. The selection criteria have been approved by the head of this thesis and consulted with the supervisor of the practical project, on which the criteria were tested for the first time. Based on these criteria, the individual manual tests are evaluated. Final count of points is used for determination of the test priority. The fourth section is devoted to calculation of the return on investment in automated testing. Because the investments made into the automated testing are very high, it must be assured that their amount will return for the selected period. The fifth part is focused on defining the principles for ensuring the proper maintenance of automated tests and describing the technical complications and frequent problems that may occur during automated testing. The last section is devoted to pilot verification of the methodology on a practical project and for determination of the test priority for automation.

O processo de automatização de teste de software possui alto custo envolvido em sistemas de larga escala, pois exigem cenários de teste complexos e tempos de execução extremamente longos. Além disso, cada etapa do processo de teste requer recursos computacionais e um tempo considerável para a execução de muitos casos de teste, tornando-se um gargalo para as empresas de Tecnologia da Informação (TI). Neste contexto, os benefícios e oportunidades oferecidos pela combinação da computação em nuvem com o Teste como Serviço (Testing as a Service, TaaS), que é considerado um novo modelo de negócio e de serviço atraente e promissor, podem proporcionar um impacto positivo na redução do tempo de execução dos testes de maneira custo-efetiva e aumentar o retorno sobre o investimento ou Return on investment (ROI). Todavia, existe o problema de vendor lock-in, que é o aprisionamento do usuário à plataforma de um fornecedor específico ou serviço de teste, ocasionado pela dificuldade de migrar de um fornecedor TaaS para outro, limitando a utilização dessas novas tecnologias de maneira efetiva e eficiente, impedindo assim, a ampla adoção do TaaS. Como os estudos existentes não são rigorosos ou conclusivos e, principalmente, devido à falta de evidência empírica na área de serviço de teste, muitas questões devem ser investigadas na perspectiva da migração entre os provedores de TaaS. O objetivo deste trabalho é reduzir o impacto ocasionado pelo problema de vendor lock-in no processo de automatização de testes na computação em nuvem, na escrita, configuração, execução e gerenciamento dos resultados de testes automatizados. Neste contexto, foi desenvolvido o protótipo da abordagem intitulada Multi-TaaS por meio de uma biblioteca Java como prova de conceito. A abordagem Multi-TaaS é uma camada de abstração e a sua arquitetura permite abstrair e flexibilizar a troca de fornecedores de TaaS de forma portável, pois permite encapsular toda a complexidade da implementação do engenheiro de software ao desacoplar o teste automatizado de qual plataforma TaaS ele será executado, bem como abstrair os aspectos da comunicação e integração entre as APIs REST proprietárias dos diferentes fornecedores de TaaS. Além disso, a abordagem Multi-TaaS possibilita também sumarizar os resultados dos testes automatizados de forma independente das tecnologias da plataforma TaaS subjacente. Foram realizadas avaliações comparativas da eficiência, efetividade, dificuldade e do esforço de migração entre as abordagens Multi-TaaS e abordagem convencional, por meio de experimentos controlados. Os resultados deste trabalho indicam que a nova abordagem permite facilitar a troca do serviço de teste, melhorar a eficiência e, principalmente, reduzir o esforço e os custos de manutenção na migração entre fornecedores de TaaS. Os estudos realizados no experimento controlado são promissores e podem auxiliar os engenheiros de software na tomada de decisão quanto aos riscos associados ao vendor lock-in no TaaS. Por fim, a abordagem Multi-TaaS contribui, principalmente, para a portabilidade dos testes automatizados na nuvem e da sumarização dos resultados dos testes e, consequentemente, possibilita que o modelo de serviço TaaS na computação em nuvem seja amplamente adotado, de forma consciente, no futuro. / The automation of software testing involves high costs in large-scale systems, since it requires complex test scenarios and extremely long execution times. Moreover, each of its steps demands computational resources and considerable time for running many test cases, which makes it a bottleneck for Information Technology (IT) companies. The benefits and opportunities offered by the combination of cloud computing and Testing as a Service (TaaS), considered a new business and service model, can reduce the execution time of tests in a cost-effective way and improve Return on Investment (ROI). However, the lock-in problem, i.e., the imprisonment of the user in the platform of a specific vendor or test service caused by the difficult migration from one TaaS provider to another limits the effective use of such new technologies and prevents the widespread adoption of TaaS. As studies conducted are neither rigorous, nor conclusive, and mainly due to the lack of empirical evidence, many issues must be investigated from the perspective of migration among TaaS providers. This research aims at reductions in the impact of the vendor lock-in problem on the automation process of testing in cloud computing, writing, configuration, execution and management of automated test results. The prototype of the Multi- TaaS approach was developed through a Java library as a proof of concept. The Multi-TaaS approach is an abstraction layer and its architecture enables the abstraction and flexibilization of the exchange of TaaS providers in a portable way, once the complexity of the software engineers implementation can be encapsulated. The two main advantages of Multi-TaaS are the decoupling of the automated test from the TaaS platform on which it will be executed and the abstraction of the communication and integration aspects among the proprietary REST APIs of the different TaaS providers. The approach also enables the summarization of automated test results independently of the underlying TaaS platform technologies. A comparative evaluation between Multi-TaaS and conventional migration approaches regarding the difficulty, efficiency, effectiveness and effort of migration among TaaS providers was conducted through controlled experiments.The results show the approach facilitates the exchange of test service, improves efficiency and reduces the effort and maintenance costs of migration among TaaS providers. The studies conducted in the controlled experiment are promising and can assist software engineers in decision-making regarding the risks associated with vendor lock-in in TaaS. The Multi-TaaS approach contributes mainly to the portability of automated tests in the cloud and summarization of their results. Finally, this research enables also the widespread adoption of the TaaS service model in cloud computing, consciously, in the future.

The project is about the virtual switch called Open vSwitch and its architecture. It deals with an acceleration of the switch mainly by using Data Plane Development Kit (DPDK). Furthermore, it describes the architecture of the DPDK kit and analyses the individual functional units. Furthermore, it describes the architecture of the DPDK kit, analyses the individual functional units and describes the possibilities of its configuration. Another part of the project describes the methodology chosen for a performance testing of virtual switches. Subsequently, this methodology was used to make a design and environment implementation for fully automatic Open vSwitch s DPDK performance testing with the use of automatic systems such as Koji, Jenkins, Beaker a VSperf. Simultaneously, the tools for automatic comparison of produced results were implemented. The created environment was then used for the performance measurement of several basic Open vSwitch configurations with and without the use of DPDK. The implemented measurements are discussed and evaluated in the project. The final project's stage provides a great amount of the enlargement and improvement of the implemented tests.