A lot of time is spent on manual software testing. When software can interact with exponentially more and more components the number of tests required will increase at the same rate. Automation of these manual tests has been previously shown that a time-saving can occur with test automation. When tests need to be done on software that interacts with physical components the results of testing automation are harder to predict. This thesis aims to explain the impact of physical components on the development and the result of test tools developed for use on automated tests on objects with physical components. In this work, a test tool for automatic regression testing for height adjustable table legs were developed. The method used to design the tool was Design Science Research where, for example, TESLA was adapted for use in automated tests with physical components. TESLA is a language for specifying test cases and allowing test automation with embedded systems that was implemented in the test tool's design which could be used to automatically generate scripts and automatically executes test cases. With the development and design of the testing tool, the thesis attempted to answer how a test tool should be designed when physical components are included in the test object and how the measurement accuracy of the tool may affect the test result. Experiments have been carried out during the development process where regression tests have been performed according to a test protocol. During these experiments a test protocol is executed. One manually by hand and one is done automatically with the testing tool. The results of the experiments show that the execution of the test protocol execution with the test tool gave a time saving of 35% compared to the manual tests. The physical components of the test object were shown to affect the automatic execution time negative and further analysis of previous research has shown that this result is insufficient to justify the large initial time that is required to automate tests. / Mycket tid spenderas ofta på manuell testning av mjukvara. Då en mjukvara kan interagera med exponentiellt mer komponenter så kommer antalet tester att öka i samma takt. Automatisering av dessa manuella tester har med tidigare forskning bevisat att en tidsbesparing kan ske med testautomatiseringen. Då tester måste utföras på mjukvara som interagerar med fysiska komponenter så är resultaten av testautomatisering inte lika klara och arbetet syftar på att förklara de fysiska komponenternas påverkan på utveckling och resultatet för testverktyg som används till automatisk testning av testobjekt med fysiska komponenter. I detta arbete så framtogs ett testverktyg för automatiska regressionstest för höj och sänkbara bordsben. Metoden som användes för att utveckla verktyget var Design Science Research där till exempel TESLA anpassades för användning i automatiska tester med fysiska komponenter. TESLA är ett språk för att specificera testfall och möjliggöra testautomatisering i inbyggda system implementerades i testverktygets design och kunde då automatisk generera skript och automatiskt exekvera testfall. Med utvecklingen och design av testverktyget så försökte arbetet svara på hur ett testverktyg bör vara designat när fysiska komponenter ingår i testobjektet och hur testverktygets mätnoggrannhet kan påverka testresultatet. Experiment har utförts under arbetets gång där regressionstester har utförts enligt ett testprotokoll. Under experimenten utfördes ett testprotokoll manuellt för hand och ett automatiskt med testverktyget. I båda situationerna så mättes testprotokollets utförande i tid. Testverktyget gav en tidsbesparing på 35% visavi de manuella testerna. Testobjektets fysiska komponenter hade en negativ påverkan på den automatiska exekveringstiden och vidare analys av tidigare forskning har visat att detta resultat inte räcker för att motivera den stora initiala tidsinvestering som krävs för att automatisera tester.

Todays’ systems for telecommunication are getting more and more complex. Automatic testing is required to guarantee quality of the systems produced. An actual example is the introduction of GPRS traffic in the GSM network nodes. This thesis investigates the need and demands for such an automatic testing of the traffic channels in the GSM system. A solution intended to be a part of the Ericsson TSS is proposed. One problem to be solved is that today’s tools for testing do not support testing of speech channels with the speech transcoder unit installed. As part of the investigation, a speech codec is implemented for execution on current hardware used in the test platform. The selected speech codec is the enhanced full rate codec, generating a bitstream of 12.2 kbit/s, and gives a good trade-off between compression and speech quality. The report covers the design of the test tool and the implementation of speech codec. Particularly performance problems in the implementation of the encoder will be addressed.

This thesis investigates the possibilities of finding solutions, in order to reduce the total time spent for testing and waiting times for running multiple automated test cases in a test framework. The “Automated Test Framework”, developed by Axis Communications AB, is used to write the functional tests to test both hardware and software of a resource. The functional tests that tests the software is considered in this thesis work. In the current infrastructure, tests are executed sequentially and resources are allocated using First In First Out scheduling algorithm. From the user’s point of view, it is inefficient to wait for many hours to run their tests that take few minutes to execute. The thesis consists of two main parts: (1) identify a plugin that suits the framework and executes the tests in parallel, which reduces the overall execution time of tests and (2) analyze various scheduling algorithms in order to address the resource allocation problem, which arose due to limited resource availability, while the tests were run in parallel. By distributing multiple tests across several resources and executing them in parallel, help in improving the test strategy, thereby reducing the overall execution times of test suites. The case studies were created to emulate the problematic scenarios in the company and sample tests were written that reflect the real tests in the framework. Due to the complexity of the current architecture and the limited resources available for running the test in parallel, a simulator was developed with the identified plugin in a multi-core computer, with each core simulating a resource. Multiple tests were run using the simulator in order to explore, check and assess if the overall execution time of the tests can be reduced. While achieving parallelism in running the automated tests, resource allocation became a problem, since limited resources are available to run parallel tests. In order to address this problem, scheduling algorithms were considered. A prototype was developed to mimic the behaviour of a scheduling plugin and the scheduling algorithms were implemented in the prototype. The set of values were given as input to the prototype and tested with scenarios described under case studies. The results from the prototype are used to analyze the impact caused by various scheduling algorithms on reducing the waiting times of the tests. The combined usage of simulator along with scheduler prototype helped in understanding how to minimize the total time spent for testing and improving the resource allocation process.

Ziel der Arbeit war es, die Systemtestautomatisierung zu vereinfachen. Gerade in Hinblick auf Stabilität und Wiederverwendbarkeit konnten die in der Praxis eingesetzten Möglichkeiten nicht vollständig überzeugen. Der in der Abhandlung erarbeitet Zielzustand, die „strukturierte Automatisierung des SystemTests“ (SAST) integriert den Systemtest in „MOdel Compiler for generating Complete Applications“ (MOCCA), ein modelgetriebenes Anwendungsgenerierungsframework. MOCCA generiert aus Struktur- und Verhaltensmodellen voll-ständige Softwaresysteme. Er wurde an der TU Bergakademie Freiberg entwickelt. Zur leichtgewichtigen Modellierung des Anwendungsverhaltens wurde es durch die Dissertation von Dr. Liang (vgl. [Lian2013]) u. a. um eine Action Language eXtended Object Constraint Language (XOCL) erweitert. Diese Verhaltensbeschreibungsmöglichkeit wurde in SAST ebenso für die Verhaltensabbildung des Systemtests genutzt und bildet einen Pfeiler in dem erstellten Prototyp zur Systemtestgenerierung. SAST bezieht sich auf GUI-basierte Softwaresysteme. Sie bildet, wie es für den Systemtest charakteristisch ist, Fachprozesse anhand der Oberfläche ab. Zur Lösung wurden, neben dem Testverhalten, Artefakte zur Teststrukturierung, Schlüsselwortbildung und eine Ausführungs-Engine erstellt und in den bestehenden Generierungspro-zess von MOCCA eingefügt. Mit den grundlegenden Charakteristika der Lösung – modellgetrieben, schlüssel-wort-orientiert und in Testfällen strukturiert – unterstützt die Arbeit die angestrebten Verbesserungen: Wiederverwendbarkeit, Wartungsarmut und frühzeitige Testfallentwicklung. Eine Ausgestaltung für konkrete Testfälle ermöglicht schnelle Testschwerpunkte und -reduzierungen im Rahmen des risikobasierten Tests.:1 Einleitung 2 Theoretisches Fundament 3 Analyse bekannter Methoden für den Systemtest 4 Strukturierte Automatisierung des SystemTests 5 Prototypische Systemtestmodellierung 6 Proof of Concept: Anwendung ,TranscriptGenerator' 7 Abschließende Bewertung und weitere Möglichkeiten

The Cloud Radio Access Network (RAN) is a groundbreaking technology employed in the telecommunications industry, offering flexible, scalable, and cost-effective solutions for seamless wireless network services. However, testing Cloud RAN applications presents significant challenges due to their complexity, potentially leading to delays and increased costs. A paramount solution to overcome these obstacles is test automation. Automating the testing process not only dramatically reduces manual efforts but also enhances testing accuracy and efficiency, expediting the delivery of high-quality products. In the current era of cutting-edge advancements, artificial intelligence (AI) and machine learning (ML) play a transformative role in revolutionizing Cloud RAN testing. These innovative technologies enable rapid identification and resolution of complex issues, surpassing traditional methods. The objective of this thesis is to adopt an AI-enabled approach to Cloud RAN test automation, harnessing the potential of machine learning and natural language processing (NLP) techniques to automatically select test cases from test instructions. Through thorough analysis, relevant keywords are extracted from the test instructions using advanced NLP techniques. The performance of three keyword extraction methods is compared, with SpaCy proving to be the superior keyword extractor. Using the extracted keywords, test script prediction is conducted through two distinct approaches: using test script names and using test script contents. In both cases, Random Forest emerges as the top-performing model, showcasing its effectiveness with diverse datasets, regardless of oversampling or undersampling data augmentation techniques. Based on the rule-based approach, the predicted test scripts are utilized to determine the order of execution among the predicted test scripts. The research findings highlight the significant impact of AI and ML techniques in streamlining test case selection and automation for Cloud RAN applications. The proposed AI-enabled approach optimizes the testing process, resulting in faster product delivery, reduced manual workload, and overall cost savings.

Test Automation is becoming a more vital part of the software development cycle, as it aims to lower the cost of testing and allow for higher test frequency. However, automating manual tests can be difficult as they tend to require complex human interaction. In this thesis, we aim to solve this by using Imitation Learning as a tool for automating manual software tests. The software under test consists of a virtual simulation, connected to a physical input device in the form of a sight. The sight can rotate on two axes, yaw and pitch, which require human motion control. Based on this, we use a Behavioral Cloning approach with a k-NN regressor trained on human demonstrations. Evaluation of model resemblance to the human is done by comparing the state path taken by the model and human. The model task performance is measured with a score based on the time taken to stabilize the sight pointing at a given object in the virtual world. The results show that a simple k-NN regression model using high-level states and actions, and with limited data, can imitate the human motion well. The model tends to be slightly faster than the human on the task while keeping realistic motion. It also shows signs of human errors, such as overshooting the object at higher angular velocities. Based on the results, we conclude that using Imitation Learning for Test Automation can be practical for specific tasks, where capturing human factors are of importance. However, further exploration is needed to identify the full potential of Imitation Learning in Test Automation.

O processo de automatização de teste de software possui alto custo envolvido em sistemas de larga escala, pois exigem cenários de teste complexos e tempos de execução extremamente longos. Além disso, cada etapa do processo de teste requer recursos computacionais e um tempo considerável para a execução de muitos casos de teste, tornando-se um gargalo para as empresas de Tecnologia da Informação (TI). Neste contexto, os benefícios e oportunidades oferecidos pela combinação da computação em nuvem com o Teste como Serviço (Testing as a Service, TaaS), que é considerado um novo modelo de negócio e de serviço atraente e promissor, podem proporcionar um impacto positivo na redução do tempo de execução dos testes de maneira custo-efetiva e aumentar o retorno sobre o investimento ou Return on investment (ROI). Todavia, existe o problema de vendor lock-in, que é o aprisionamento do usuário à plataforma de um fornecedor específico ou serviço de teste, ocasionado pela dificuldade de migrar de um fornecedor TaaS para outro, limitando a utilização dessas novas tecnologias de maneira efetiva e eficiente, impedindo assim, a ampla adoção do TaaS. Como os estudos existentes não são rigorosos ou conclusivos e, principalmente, devido à falta de evidência empírica na área de serviço de teste, muitas questões devem ser investigadas na perspectiva da migração entre os provedores de TaaS. O objetivo deste trabalho é reduzir o impacto ocasionado pelo problema de vendor lock-in no processo de automatização de testes na computação em nuvem, na escrita, configuração, execução e gerenciamento dos resultados de testes automatizados. Neste contexto, foi desenvolvido o protótipo da abordagem intitulada Multi-TaaS por meio de uma biblioteca Java como prova de conceito. A abordagem Multi-TaaS é uma camada de abstração e a sua arquitetura permite abstrair e flexibilizar a troca de fornecedores de TaaS de forma portável, pois permite encapsular toda a complexidade da implementação do engenheiro de software ao desacoplar o teste automatizado de qual plataforma TaaS ele será executado, bem como abstrair os aspectos da comunicação e integração entre as APIs REST proprietárias dos diferentes fornecedores de TaaS. Além disso, a abordagem Multi-TaaS possibilita também sumarizar os resultados dos testes automatizados de forma independente das tecnologias da plataforma TaaS subjacente. Foram realizadas avaliações comparativas da eficiência, efetividade, dificuldade e do esforço de migração entre as abordagens Multi-TaaS e abordagem convencional, por meio de experimentos controlados. Os resultados deste trabalho indicam que a nova abordagem permite facilitar a troca do serviço de teste, melhorar a eficiência e, principalmente, reduzir o esforço e os custos de manutenção na migração entre fornecedores de TaaS. Os estudos realizados no experimento controlado são promissores e podem auxiliar os engenheiros de software na tomada de decisão quanto aos riscos associados ao vendor lock-in no TaaS. Por fim, a abordagem Multi-TaaS contribui, principalmente, para a portabilidade dos testes automatizados na nuvem e da sumarização dos resultados dos testes e, consequentemente, possibilita que o modelo de serviço TaaS na computação em nuvem seja amplamente adotado, de forma consciente, no futuro. / The automation of software testing involves high costs in large-scale systems, since it requires complex test scenarios and extremely long execution times. Moreover, each of its steps demands computational resources and considerable time for running many test cases, which makes it a bottleneck for Information Technology (IT) companies. The benefits and opportunities offered by the combination of cloud computing and Testing as a Service (TaaS), considered a new business and service model, can reduce the execution time of tests in a cost-effective way and improve Return on Investment (ROI). However, the lock-in problem, i.e., the imprisonment of the user in the platform of a specific vendor or test service caused by the difficult migration from one TaaS provider to another limits the effective use of such new technologies and prevents the widespread adoption of TaaS. As studies conducted are neither rigorous, nor conclusive, and mainly due to the lack of empirical evidence, many issues must be investigated from the perspective of migration among TaaS providers. This research aims at reductions in the impact of the vendor lock-in problem on the automation process of testing in cloud computing, writing, configuration, execution and management of automated test results. The prototype of the Multi- TaaS approach was developed through a Java library as a proof of concept. The Multi-TaaS approach is an abstraction layer and its architecture enables the abstraction and flexibilization of the exchange of TaaS providers in a portable way, once the complexity of the software engineers implementation can be encapsulated. The two main advantages of Multi-TaaS are the decoupling of the automated test from the TaaS platform on which it will be executed and the abstraction of the communication and integration aspects among the proprietary REST APIs of the different TaaS providers. The approach also enables the summarization of automated test results independently of the underlying TaaS platform technologies. A comparative evaluation between Multi-TaaS and conventional migration approaches regarding the difficulty, efficiency, effectiveness and effort of migration among TaaS providers was conducted through controlled experiments.The results show the approach facilitates the exchange of test service, improves efficiency and reduces the effort and maintenance costs of migration among TaaS providers. The studies conducted in the controlled experiment are promising and can assist software engineers in decision-making regarding the risks associated with vendor lock-in in TaaS. The Multi-TaaS approach contributes mainly to the portability of automated tests in the cloud and summarization of their results. Finally, this research enables also the widespread adoption of the TaaS service model in cloud computing, consciously, in the future.

Regression testing is just as important for the quality assurance of a system, as it is time consuming. Several techniques exist with the purpose of lowering the execution times of test suites and provide faster feedback to the developers, examples are ones based on transition-models or string-distances. These techniques are called test case selection (TCS) techniques, and focuses on selecting subsets of the test suite deemed relevant for the modifications made to the system under test. This thesis project focused on evaluating the use of a topic model, latent dirichlet allocation, as a means to create a diverse selection of test cases for coverage of certain test characteristics. The model was tested on authentic data sets from two different companies, where the results were compared against prior work where TCS was performed using similarity-based techniques. Also, the model was tuned and evaluated, using an algorithm based on differential evolution, to increase the model’s stability in terms of inferred topics and topic diversity. The results indicate that the use of the model for test case selection purposes was not as efficient as the other similarity-based selection techniques studied in work prior to thist hesis. In fact, the results show that the selection generated using the model performs similar, in terms of coverage, to a randomly selected subset of the test suite. Tuning of the model does not improve these results, in fact the tuned model performs worse than the other methods in most cases. However, the tuning process results in the model being more stable in terms of inferred latent topics and topic diversity. The performance of the model is believed to be strongly dependent on the characteristics of the underlying data used to train the model, putting emphasis on word frequencies and the overall sizes of the training documents, and implying that this would affect the words’ relevance scoring to the better.

<p>The products The Company manufactures are used in an area of the industry where errors leading to a stop in production can be quite expensive. Therefore are testing of the products important and the tests can also give indications about the quality of the products.</p><p>The Company is in a phase where they are developing a new product line to support all existing and future products. In this phase, it was decided that all products will use a common framework for unit testing and a common build system for all products. One part of the thesis was to investigate and evaluate different frameworks for unit testing and tools for a build system. The unit test framework that were evaluated are Cppunit, cfix, NUnit, Boost test library, Unit Testing++ and CxxTest. The result of the evaluation was that CppUnit were recommended. For the build system MSBuild, NAnt, Automated Build Studio and Cruise Control .NET were evaluated. The recommended tools for a build system is MSBuild in combination with Cruise Control .Net if The Company is interested in the functionality Cruise Control .Net has to offer.</p><p>The Company also has an interest in evaluating the current test processes and identify improvements as a part of The Company’ s objective were all products should follow a common test process. In order to identify these improvements a literature study of four test process improvement frameworks (Test Process Improvement, Test Maturity Model Integrated, Minimal Test Practice Framework and Test Improvement Model) were carried out. Out of these four frameworks Test Process Improvement (TPI) were chosen to assist when identifying improvements. With the help of TPI a limited assessment took place to give indications about the test maturity for three of The Company’s products where two of the products had low maturity. Results of the improvement measures can be summed up with the need to harmonize the documents, standardize and document the various processes.</p><p>As a last part of the thesis the possibility to automate testing of two of the products graphical user interfaces with the program Test Complete were investigated. For one of the products the result was that it worked satisfactorily and for the other product it did not work at all. This resulted in recommendations for how The Company should proceed with automated testing of the graphical user interface.</p> / <p>De produkter som företaget tillverkar används i huvudsak inom ett område av branschen där fel som leder till stopp i produktionen kan vara ganska dyrt. Detta gör testning av produkterna viktiga och tester kan också ge indikationer om kvaliteten på produkterna.</p><p>Företaget är i en fas där man utvecklar en ny produktlinje som ska stödja alla befintliga och framtida produkter. I denna fas har man beslutat att alla produkterna ska använda ett gemensant ramverk för enhetstestning och även använda ett gemensamt byggsystem för samtliga produkter. En del av examensarbetet var att undersöka och utvärdera olika ramverk för enhets testning och verktyg för byggsystem. De ramverk som utvärderades var CppUnit, cfix, NUnit, Boost test library, unitTest++ och CxxTest. Utvärderingen ledde fram till att CppUnit rekommenderades till företaget. Verktyg som utvärderades för byggsystem var MSBuild, NAnt, Automated Build Studio och Cruise Control .Net. För byggsystem rekommenderas MSBuild i kombination med Cruise Control .Net ifall företaget är intresserade av den extra funktionalitet som Cruise Control .Net har att erbjuda.</p><p>Företaget har även ett intresse av att utvärdera den nuvarande testprocessen och identifiera förbättringar som ett led i att befintliga produkter skall följa en gemensam testprocess. För att kunna identifiera dessa förbättringar utfördes en litteraturstudie över fyra stycken test process förbättrings ramverk (Test Process Improvent, Test Maturity Model integrated, Minimal Test Practice Framework och Test Improvement Model). Utav dessa fyra ramverk så valdes Test Process Improvement (TPI) ut som en hjälp för att identifiera förbättringar. Med hjälp av TPI utfördes en begränsad bedömning av företagets mogenhetsgrad på tre produkter, där två av produkterna har en låg mogenhetsgrad. Resultat av förbättringsåtgärderna kan sammanfattas med att man bör harmonisera dokument, standardisera och dokumentera olika processer.</p><p>Som en sista del i examensarbetet så utvärderades möjligheten att automatisera testning på två av produkternas grafiska användargränssnitt med programmet TestComplete. För en av produkterna blev resultatet att det fungerade tillfredställande och för den andra produkten så fungerade det inte alls. Resultatet blev rekommendationer för hur företaget borde gå vidare med automatisering av testnig på det grafiska användargränssnittet.</p>

As the complexity of software system increases, delivering quality software successfully becomes an ever more challenging task. Applying automated testing techniques effectively to the software development process can reduce software testing effort substantially and assure software quality cost-effectively. Thus the future of software testing will rely heavily on automated testing techniques. This thesis describes a practical approach to automated software testing by investigating and analyzing different automation test tools in real-world situations. In view of the fact that the key to successful automated testing is planning, understanding the requirements for automated testing and effectively planning is critical and essential. iv This thesis presents the design and implementation of an automated testing framework. It consists of an automated testing tool, which is based on the commercial product TestComplete, as well as associated testing processes. The application area is telephony communications software. To demonstrate the viability of our automated testing approach, we apply our testing framework to a Voice-over-IP telephony application called Desktop Assistant. This case study illustrates the benefits and limitations of our automated testing approach effectively.