Recent advances in test automation have seen a host of new techniques for automated test generation, which traditionally has largely been a manual and expensive process. These techniques have enabled generation of much larger numbers of tests at a much reduced cost. When executed successfully, these tests enable a significant increase in our confidence in the program's correctness. However, as our ability to generate greater numbers of tests increases, we are faced with the problem of the likely high cost of executing all the tests in terms of the total execution time. This thesis presents a novel approach - clustered test execution - to address this problem. Instead of executing each test case separately, we execute parts of several tests using a single execution, which then forks into several directions as the behaviors of the tests differ. Our insight is that in a large test suite, several tests are likely to have common initial execution segments, which do not have to be executed over and over again; rather such a segment could be executed once and the execution result shared across all those tests. As an enabling technology we use the Java PathFinder(JPF) model checker, which is a popular explicit-state model checker for Java programs. Experimental results show that our clustering approach for test execution using JPF provides speed-ups over executing each test in turn from a test suite on the JPF java virtual machine. / text

Automated software test execution is a critical part of the modern software development process, where rapid feedback on the product quality is expected. It is of high importance that impediments related to test execution automation are prevented and removed as quickly as possible. An enabling factor for all types of improvement is to understand the nature of what is to be improved. The goal with this thesis is to further the knowledge about common problems encountered by software developers using test execution automation, in order to enable improvement of test execution automation in industrial software development. The research has been performed through industrial case studies and literature reviews. The analysis of the data have primarily been performed using qualitative methods, searching for patterns, themes, and concepts in the data.  Our main findings include: (a) a comprehensive list of potential impediments reported in the published body of knowledge on test execution automation, (b) an in-depth analysis of how such impediments may affect the performance of a development team, and (c) a proposal for a qualitative model of interactions between the main groups of phenomena that contribute to the formation of impediments in a test execution automation project. In addition to this, we contribute qualitative and quantitative empirical data from our industrial case studies.  Through our results, we find that test execution automation is a commonly under-estimated activity,  not only in terms of resources but also in terms of the complexity of the work. There is a clear tendency to perform the work ad hoc, down-prioritize the automation in favor of other activities,  and ignore the long-term effects in favor of short-term gains. This is both a technical and a cultural problem that need to be managed by awareness of the problems that may arise, and also have to be solved in the long term through education and information. We conclude by proposing a theoretical model of the socio-technical system that needs to be managed to be successful with test execution automation. / Syftet med denna avhandling är att utöka den vetenskapliga kunskapen om problem som kan uppstå under användning av automatiserad testning i industriell programvaruutveckling. Utveckling av programvara består principiellt av ett antal processteg: kravbehandling, detaljerad systemkonstruktion, implementation i form av programmering, och slutligen testning som säkerställer att kvaliteten på programvaran är tillräcklig för dess tilltänkta användare. Testning representerar en stor del av tiden och kostnaden för utveckling av programvaran, och detta gör att det är attraktivt att automatisera testningen. Automatiserade tester kan bidra med många positiva effekter. Testning som utförs om och om igen, för att säkerställa att gammal funktionalitet inte slutar att fungera när ändringar görs i programvaran, kan med fördel göras automatiserat. Detta frigör kvalificerad personal till kvalificerat arbete. Automatisering kan även minska ledtiden för testningen och därmed möjliggöra snabbare leveranser till kund. Ett annat mål med testautomatisering är att vara säker på att samma tester utförs på ett likartat sätt varje gång produkten testas, så att den håller en jämn och stabil kvalitetsnivå. Automatiserad testning är dock en mer komplex och kostsam verksamhet än vad man kan tro, och problem som uppstår under dess användning kan ha stora konsekvenser. Detta gäller i ännu större utsträckning i organisationer som använder moderna utvecklingsmetoder där automatisering är grundstenen för en effektiv kvalitetskontroll. För att kunna undvika så många problem som möjligt, är det därför mycket viktigt att förstå vad som händer när man använder testautomatisering i stor skala. Denna avhandling presenterar resultat från fallstudier i svensk industri, som, kombinerat med en systematisk genomgång av befintlig forskning inom området, har utförts för att söka djupare kunskap och möjligheter till generalisering. Arbetet har varit beskrivande, och förklarande, och bygger på empirisk forskningsmetodik.  I avhandlingen bidrar vi med (a) information om de problem relaterade till automatiserad testning som vi har identifierat i de empiriska fallstudierna, (b) en diskussion av dessa problem i relation till andra studier i området, (c) en systematisk litteraturstudie som ger en översikt över relevanta publikationer i området, (d) en analys av bevisen som insamlats genom litteraturstudien, samt (e) en modell av det organisatoriska och tekniska system som måste hanteras för att man skall nå framgång med ett testautomatiseringsprojekt. Resultaten tyder på att rent tekniska problem inte utgör huvuddelen av de problem som upplevs med testautomatisering. Istället handlar det till stora delar om organisationell dynamik: hantering av förändringen det innebär att införa automatisering, planering av automatisering och dess användning, samt vilka finansiella förväntningar man har på automatiseringen. / ITS-EASY Post Graduate School for Embedded Software and Systems

Context: Testing is an important activity in software development and is typically estimated to account for nearly half of the efforts in the software development cycle. This puts a great demand on improving the artifacts involved in this task such as the test cases and test suites (a collection of test cases).  Objective: When executing test programs, it is typical to record runtime information associated with the test cases in the form of test execution logs or traces. The aim of this work is to explore how this information can be utilized to improve the software testing process. To this end, two main aspects are investigated which are (1) in the context of test case generation and (2) in the context of different optimizations regarding existing test suites. Furthermore, the role of the logs regarding fault localization in connection with improving the existing test suites is investigated. Method: A systematic literature review is conducted to investigate, identify and analyze the existing literature on test case generation and test suite optimization that utilizes the test execution logs. Results: After a rigorous search in six digital databases, 26 primary studies were identified. 5 of the selected papers propose approaches in the context of test data generation, 8 papers suggest test case prioritization (TCP) techniques, 4 papers discuss approaches in test case selection (TCS), and 5 papers propose approaches in test suite minimization (TSM). Furthermore, we identified, 3 papers that discuss fault localization, and one paper that discussed the decomposition of large test cases into smaller single purpose test cases using the logs from previous test executions. Conclusion: The test execution logs are a useful source of information for different testing activities. Regarding test case generation, the main theme observed is the use of genetic algorithms in attempting to generate appropriate test cases when the alternative might have been to use random test data generation methods. When it comes to improving existing test suites several approaches within TCP, TCS and TSM such as similarity-based, modification-based, cluster-based, and search-based were put forward by the authors of the selected primary studies. Furthermore, several fault localization techniques using the logs were suggested.

Made available in DSpace on 2014-06-12T15:49:48Z (GMT). No. of bitstreams: 1 license.txt: 1748 bytes, checksum: 8a4605be74aa9ea9d79846c1fba20a33 (MD5) Previous issue date: 2009 / Em mercados competitivos como, por exemplo, o de celulares, empresas de software que liberam produtos com baixa qualidade podem rapidamente perder os seus clientes. A fim de evitar esse problema, essas empresas devem garantir que a qualidade dos produtos atenda a expectativa de seus clientes. Nesse contexto, testes é uma das atividades mais utilizadas para se tentar melhorar a qualidade de um software. Além disso, o resultado da atividade de teste está sendo considerado tão importante que em muitos casos é preferível alocar equipes exclusivamente para exercer atividades de teste. Essas equipes de teste devem ser capazes de estimar o esforço exigido para exercer as suas atividades dentro do prazo ou para solicitar mais recursos ou negociar prazos quando necessário. Na prática, as consequências de se ter estimativas ruins são onerosas para a organização: redução de escopo, atraso nas entregas ou horas extras de trabalho. O impacto dessas consequências é ainda maior em se tratando de execução manual de testes. Visando uma melhor forma de estimar esforço de execução manual de casos de teste funcionais, esta pesquisa propõe e valida uma medida de tamanho de teste e de complexidade de execução baseada nas próprias especificações dos testes, bem como um método de medição para a métrica proposta. Além disso, diversos estudos de caso, survey e experimentos foram realizados para avaliar o impacto desse trabalho. Durante esses estudos, verificamos uma melhoria significativa proporcionada por nossa abordagem na precisão das estimativas de esforço de execução de testes manuais. Também identificamos fatores de custo relacionados a atividades de execução manual de testes utilizando julgamento de especialistas. O efeito desses fatores foram investigados através da execução de experimentos controlados, onde pudemos constatar que apenas alguns dos fatores identificados tiveram efeito significativo. Por fim, diversas ferramentas de suporte foram desenvolvidas durante essa pesquisa, incluindo a automação das estimativas de esforço de execução de testes a partir de especificações de testes escritas em linguagem natural

Inom programvaruutveckling är testning en viktig aktivitet där regressionstest är centralt. Regressionstestning innebär att exekvera om testfall för att säkerställa att de förändringarna som har gjorts i programmet inte har infört nya fel i systemet. Detta görs också med syfte till att kontrollera att systemets befintliga funktioner inte har påverkats negativt av dessa förändringar. Att genomföra regressionstester manuellt är resurskrävande i form av tid och pengar. Därför rekommenderas det att utföra denna aktivitet på ett automatiserat sätt. Ett problem som finns idag med exekvering av automatiserad regressionstest är att testare blir specialiserade på specifika testmiljöer på grund av det används många olika testmiljöer. Därför skulle de inte kunna stödja förbättringen av andra testares arbetsflöde. Således fokuserar denna studie på att beskriva de aktuella arbetssätt inom området exekvering av automatiserade regressionstester samt hur dessa kan utvecklas. Det har varit nödvändigt att genomföra personliga intervjuer samt litteraturstudie för att kunna besvara studiens forskningsfrågor. I detta arbete har det kommits fram till skapandet av en allmän ram för testare att arbeta mer effektivt med genomförandet av automatiserade regressionstestning oavsett testmiljön. Ramverket innehåller 7 faser: (1) Val av testfall, (2) Utföring/Exekvering av tester, (3) Kontroll/Analysera av resultat, (4) Skrivandet av rapport för testresultat, (5) Arkivering av testresultat rapporter, (6) Värdering och tilldelning av uppkommande problem samt (7) Hantering av testcykeln. / Software testing is an important activity within the software development area, where regression testing is essential. Regression testing implies re-running test cases in order to ensure that changes made to the software do not introduce new errors and to guarantee that the system’s functionalities have not been affected by those changes. To execute regression testing in manual mode involves valuable resources, specifically time and money. Therefore it is recommended to carry out this activity in an automated manner. A problem that faces the execution of automated regression testing is that testers are becoming specialized in specific test environments due to the existing diversity of tools used. Therefore testers would not be able improve test processes related to different environments. Thus, this paper focuses on identifying the current working manner within the domain of execution of automated regression testing and to improve it. Then, it has been necessary to conduct personal interviews (7) and a literature study in order to answer the study’s questions. This work provides a general framework for testers to work more effectively with execution of automated regression testing regardless of the test environment. The framework includes 7 stages: (1) Select test cases, (2) Execute/Run tests, (3) Monitoring/Analyzing results, (4) Write a report of the test results, (5) Archive the reports of the test results, (6) Estimate and assign the raised problems, and (7) Manage the test cycle.

In recent years, automotive engineering becomes more challenging and complex than previous years because of the innovation that transformed the cars powertrain system. By the quick developing of the automotive systems and increasing the consumer demand for Hybrid Vehicles (HV), Electric Vehicles (EV) and Autonomous cars, authenticity and security of engine systems are an essential part. The Hybrid Electric Vehicle structure contained several system sections like Electric Motor, Inverter, DC-DC Converter, etc. For testing this device, OEM’s are providing a special automotive standard norm (these norms are individually defining by automotive companies as LV124/123/148) which used for simulates variety of the electrical disturbances and monitoring the system behavior. This standard profile applies, simulates and records by EM Autowave system. Electrical tests have been done for a long time manually. Manual testing takes a huge amount of time, money and human resources. Based on the facts, the thesis purpose is to provide the automated system test for supporting test case execution on the HEV test bench, and offline classification independent of the test bench. This scope of the project will be developed by adapting automation library which is going to equip new features for automation system tool. On the other hand, in order to conform the test simulator and their communication system, a device file is implemented for loading and testing the electrical profiles on the test bench system. In addition, automated classification and evaluation used Matlab functions and the test result will be provided as plots which makes differential discovering very easier.