Despite much scepticism and problems for its adoption, the Model-Driven Development (MDD) is being used and improved to provide many inherent benefits for industry. One of its greatest benefits is the ability to handle the complexity of software development by raising the abstraction level. Models are expressed using concepts that are not related to a specific implementation technology (e.g. Unified Modelling Language -UML, Object Constraint Language -OCL, Action Language for Foundational UML -ALF), which means that the models can be easier to specify, maintain and document. As in Model-Driven Engineering (MDE), the primary artefacts are the conceptual models, efforts are focused on their creation, testing and evolution at different levels of abstraction through transformations because if a conceptual schema has defects, these are passed on to the following stages, including coding. Thus, one of the challenges for researchers and developers in Model-Driven Development is being able to identify defects early on, at the conceptual schema level, as this helps reduce development costs and improve software quality. Over the last decade, little research work has been performed in this area. Some of the causes of this are the high theoretical complexity of testing conceptual schemas and the lack of adequate software support. This research area thus admits new methods and techniques, facing challenges such as generation of test cases using information external to the conceptual schemas (i.e. requirements), the measurement of possible automation, selection and prioritization of test cases, the need for an efficient support tool using standard semantics, the opportune feedback to support the software quality assurance process and facilitate making decisions based on the analysis and interpretation of the results. The aim of this thesis is to mitigate some of the problems that affect conceptual schema validation by providing a novel testing-based validation framework based on Model-Driven Development. The use of MDD improves abstraction, automation and reuse, which allows us to alleviate the complexity of our validation framework. Furthermore, by leveraging MDD techniques (such as metamodeling, model transformations, and models at runtime), our framework supports four phases of the testing process: test design, test case generation, test case execution and the evaluation of the results. In order to provide software support for our proposal, we developed the CoSTest ALF-based testing environment. To ensure that CoSTest offers the necessary functionality, we first identified a set of functional requirements. Then, after these requirements were identified, we defined the architecture and testing environment of the validation framework, and finally we implemented the architecture in the Eclipse context. CoSTest has been developed to test several properties on the executable model, such as syntactic correctness (i.e. all the elements in the model conform to the syntax of the language in which it is described), consistency between the structural and behavioural parts (its integrity constraints) and completeness (i.e. all possible changes on the system state can be performed through the execution of the operations defined in the executable model). For defective models, the CoSTest report returns a meaningful feedback that helps locate and repair any defects detected. / A pesar del escepticismo y dificultades en su adopción, el Desarrollo Orientado por Modelos (MDD, por sus siglas en inglés) está siendo usado y mejorado para proveer muchos beneficios inherentes a la industria. Uno de sus mayores beneficios es la capacidad de manejar la complejidad del desarrollo de software elevando el nivel de abstracción. Los modelos se expresan utilizando conceptos que no están relacionados con una tecnología de implementación específica (por ejemplo, Lenguaje de Modelado Unificado -UML, Lenguaje de Restricción de Objetos -OCL, Lenguaje de Acción para el Foundational UML - ALF), lo que significa que los modelos pueden ser más fáciles de especificar, mantener y documentar. Debido a que en una Ingeniería dirigida por modelos (MDE), los artefactos primarios son los modelos conceptuales, los esfuerzos se centran en su creación, prueba y evolución a diferentes niveles de abstracción a través de transformaciones, porque si un esquema conceptual tiene defectos, éstos se pasan a las siguientes etapas, incluida la codificación. Por lo tanto, uno de los retos para los investigadores y desarrolladores in MDD es poder identificar los defectos temprano, a nivel de esquemas conceptuales, ya que esto ayudaría a reducir los costos de desarrollo y mejorar la calidad del software. Durante la última década, pocos trabajos de investigación se han realizado en esta área. Algunas de las causas de esta realidad son la alta complejidad teórica de probar esquemas conceptuales y la falta de soporte de software adecuado. Por lo tanto, este área de investigación admite nuevos métodos y técnicas, enfrentando retos como la generación de casos de prueba utilizando información externa a los esquemas conceptuales (es decir, los requisitos), la medición de una posible automatización, selección y priorización de casos de prueba, la necesidad de una herramienta de soporte eficiente que utilice una semántica estándar, la retroalimentación oportuna para apoyar el proceso de aseguramiento de la calidad del software y facilitar la toma de decisiones basadas en el análisis y la interpretación de los resultados. El objetivo de esta tesis es mitigar algunos de los problemas que afectan la validación de los esquemas conceptuales, proporcionando un nuevo marco de validación basado en pruebas que fue construido usando un desarrollo dirigido por modelos. El uso de MDD permite un aumento en la abstracción, automatización y reutilización que nos permite aliviar la complejidad de nuestro marco de validación. Además, al aprovechar las técnicas MDD (como el metamodelado, las transformaciones de modelos y los modelos en tiempo de ejecución), nuestro marco soporta cuatro fases del proceso de prueba: diseño de pruebas, generación de casos de prueba, ejecución de casos de prueba y la evaluación de los resultados. Con el fin de proporcionar soporte de software para nuestra propuesta, hemos desarrollado CoSTest, un entorno de pruebas basado en el lenguaje ALF. Para asegurar que CoSTest ofrece la funcionalidad necesaria, primero identificamos un conjunto de requisitos funcionales. Luego, después de identificar estos requisitos, definimos la arquitectura y el ambiente de pruebas de nuestro marco de validación y, finalmente, implementamos la arquitectura en el contexto de Eclipse. CoSTest ha sido desarrollado para probar varias propiedades sobre el modelo ejecutable como la corrección sintáctica (es decir, todos los elementos del modelo se ajustan a la sintaxis del lenguaje en el que se describe), consistencia entre la parte estructural y el comportamiento (sus restricciones de integridad) y completitud (es decir, todos los cambios posibles en el estado del sistema se pueden realizar a través de la ejecución de las operaciones definidas en el modelo ejecutable). Para los modelos defectuosos, el informe de CoSTest devuelve una retroalimentación significativa que ayuda a localizar y reparar los defectos detec / A pesar de l'escepticisme i les dificultats en la seua adopció, el Desenvolupament Orientat per Models (MDD, segons les sigles en anglès) està sent usat i millorat per tal de proveir molts beneficis potencials inherents a l' indústria. Un dels majors beneficis és la capacitat de manejar la complexitat del desenvolupament del programari elevant el nivell d'abstracció. Els models s'expressen mitjançant conceptes que no estan relacionats amb una tecnologia d'implementació específica (per exemple, el Llenguatge de Modelat Unificat - UML, Llenguatge de Restricció d'Objectes -OCL, Llenguatge d'Acció per al Foundational UML - ALF), el que significa que els models poder ser més fàcils d'especificar, mantindre i documentar. A causa de que en una Enginyeria dirigida per models (MDE), els artefactes primaris són els models conceptuals, els esforços es centren en la seua creació, prova i evolució a diferents nivells d'abstracció mitjançant transformacions, perquè si un esquema conceptual té defectes, aquestos es passen a les següents etapes, inclosa la codificació. Per tant, un del reptes per als investigadors i desenvolupadors en MDD és poder identificar els defectes des del principi, a nivell de esquemes conceptuals, perquè açò ajudaria a reduir els costos de desenvolupament i millora de la qualitat del programari. Durant l'última dècada, pocs treballs d'investigació s'han fet en aquesta àrea. Algunes de les causes d'aquesta realitat són l'alta complexitat teòrica de provar esquemes conceptuals i la falta de suport de programari adequat. Per tant, aquesta àrea d'investigació admet nous mètodes i tècniques, enfrontant reptes com la generació de casos de prova mitjançant informació externa als esquemes conceptuals (es a dir, requisits), la medició de una possible automatització, selecció i priorització de casos de prova, la necessitat de una ferramenta de suport rentable que utilitze una semàntica estàndard, la retroalimentació oportuna per suportar el procés d'assegurament de la qualitat del programari i la facilitat per a prendre decisions basades en l'anàlisi i la interpretació dels resultats. En aquesta tesi intentem mitigar alguns dels problemes que afecten a la validació dels esquemes conceptuals, proporcionant un nou marc de validació basat en proves que va ser construït mitjançant un desenvolupament dirigit per models. L'ús de MDD permet un augment en l'abstracció, automatització i reutilització que ens permet alleujar la complexitat del nostre marc de validació. A més a més, al aprofitar les tècniques MDD (com el metamodelat, les transformacions de models i els models en temps d'execució), el nostre marc suporta quatre fases del procés de prova: disseny, generació i execució de casos de prova, així com l'avaluació de resultats del procés de prova. Amb la finalitat de proporcionar suport de programari per a la nostra proposta, hem desenvolupat un entorn de proves basat en el llenguatge ALF que s'anomena CoSTest. Per tal d'assegurar que CoSTest ofereix la funcionalitat necessària, identifiquem un conjunt de requisits funcionals abans de desenvolupar la ferramenta. Després d'identificar aquestos requisits, definim l'arquitectura i l'ambient de proves del nostre marc de validació, i finalment, implementem l'arquitectura en el context Eclipse. CoSTest ha sigut desenvolupat per provar diverses propietats sobre el model executable com la correcció sintàctica (és a dir, tots els elements del model s'ajusten a la sintaxi del llenguatge en el que es descriu), consistència antre la part estructural i el comportament (les seues restriccions d'integritat) i completitud (és a dir, tots els canvis possibles en l'estat del sistema es poden realitzar mitjançant l'execució de les operacions definides en el model executable). Per als models defectuosos, l'informe de CoSTest retorna una retroalimentació significativa que ajuda a localitzar i reparar els defectes dete / Granda Juca, MF. (2017). Testing-Based Conceptual Schema Validation in a Model-Driven Environment [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/89091

Thesis (MSc)--Stellenbosch University, 2013. / ENGLISH ABSTRACT: In this dissertation we present a software tool, Artemis, that symbolically executes Java virtual machine bytecode to find bugs and automatically generate test cases to trigger the bugs found. Symbolic execution is a technique of static software analysis that entails analysing code over symbolic inputs—essentially, classes of inputs—where each class is formulated as constraints over some input domain. The analysis then proceeds in a path-sensitive way adding the constraints resulting from a symbolic choice at a program branch to a path condition, and branching non-deterministically over the path condition. When a possible error state is reached, the path condition can be solved, and if soluble, value assignments retrieved to be used to generate explicit test cases in a unit testing framework. This last step enhances confidence that bugs are real, because testing is forced through normal language semantics, which could prevent certain states from being reached. We illustrate and evaluate Artemis on a number of examples with known errors, as well as on a large, complex code base. A preliminary version of this work was successfully presented at the SAICSIT conference held on 1–3 October 2012, in Centurion, South Africa. / AFRIKAANSE OPSOMMING: In die dissertasie bied ons ’n stuk sagtewaregereedskap, Artemis, aan wat biskode van die Java virtuele masjien simbolies uitvoer om foute op te spoor en toetsgevalle outomaties voort te bring om die foute te ontketen. Simboliese uitvoering is ’n tegniek van statiese sagteware-analise wat behels dat kode oor simboliese toevoere—in wese, klasse van toevoer—geanaliseer word, waar elke klas geformuleer word as beperkinge oor ’n domein. Die analise volg dan ’n pad-sensitiewe benadering deur die domeinbeperkinge, wat volg uit ’n simboliese keuse by ’n programvertakking, tot ’n padvoorwaarde by te voeg en dan nie-deterministies vertakkings oor die padvoorwaarde te volg. Wanneer ’n moontlike fouttoestand bereik word, kan die padvoorwaarde opgelos word, en indien dit oplaasbaar is, kan waardetoekennings verkry word om eksplisiete toetsgevalle in ’n eenheidstoetsingsraamwerk te formuleer. Die laaste stap verhoog vertroue dat die foute gevind werklik is, want toetsing word deur die normale semantiek van die taal geforseer, wat sekere toestande onbereikbaar maak. Ons illustreer en evalueer Artemis met ’n aantal voorbeelde waar die foute bekend is, asook op ’n groot, komplekse versameling kode. ’n Voorlopige weergawe van die´ werk is suksesvol by die SAICSIT-konferensie, wat van 1 tot 3 Oktober 2012 in Centurion, Suid-Afrika, gehou is, aangebied.

<p>Testing is the dominant verification technique used in the software industry today. The use of automatic test case execution increases, but the creation of test cases remains manual and thus error prone and expensive. To automate generation and selection of test cases, model-based testing techniques have been suggested.</p><p>In this thesis two central problems in model-based testing are addressed: the problem of how to formally specify coverage criteria, and the problem of how to generate a test suite from a formal timed system model, such that the test suite satisfies a given coverage criterion. We use model checking techniques to explore the state-space of a model until a set of traces is found that together satisfy the coverage criterion. A key observation is that a coverage criterion can be viewed as consisting of a set of items, which we call coverage items. Each coverage item can be treated as a separate reachability problem. </p><p>Based on our view of coverage items we define a language, in the form of parameterized observer automata, to formally describe coverage criteria. We show that the language is expressive enough to describe a variety of common coverage criteria described in the literature. Two algorithms for test case generation with observer automata are presented. The first algorithm returns a trace that satisfies all coverage items with a minimum cost. We use this algorithm to generate a test suite with minimal execution time. The second algorithm explores only states that may increase the already found set of coverage items. This algorithm works well together with observer automata.</p><p>The developed techniques have been implemented in the tool CoVer. The tool has been used in a case study together with Ericsson where a WAP gateway has been tested. The case study shows that the techniques have industrial strength.</p>

Naujuose programiniuose produktuose plačiai taikoma mobilioji įranga (mobilieji telefonai, delniniai kompiuteriai). Programinės įrangos kūrimo procese svarbią vietą užima testavimas. Dabartinių mobiliųjų įranginių apribojimai (darbo sparta, atminties kiekis, energija, ekrano dydis, platformų įvairumas) kelia naujas problemas programinės įrangos kūrimo procesui, tame tarpe ir testavimui. Testavimo proceso automatizavimas leidžia sumažinti bandymų trukmę, padidinti testavimo darbų apimtį. Programinės įrangos testavime mobiliems įrenginiams svarbu išskirti grafinės vartotojo sąsajos testavimą. Svarbi testavimo dalis – testavimas pagal specifikaciją. Šiame darbe nagrinėjamas grafinės vartotojo sąsajos automatizuotas testavimas mobiliai įrangai, nustatomi tikslai, galimi keliai jiems pasiekti bei galimas testavimo aplinkos praplėtimas testų generavimui pagal specifikaciją. / Mobile devices such as cell phones and personal digital assistance are widely used in new software products. Testing takes important place in software development process. Constraints of mobile devices (speed, amount of memory, energy, small screen, wide range of platforms) raise new problems for software development process including testing phase. Automated approach of software testing reduces testing time and increases testing range. It is important to distinguish graphic user interface as a special part of testing. The important part of testing is specification based testing. The aim of these master theses is to analyze automated testing of GUI for mobile devices, define testing tasks and enhance mobile device testing framework by providing means for test case generation from specification.

Verification and validation is one of the most important concerns in the area of software engineering towards more reliable software development. Hence it is important to overcome the challenges of testing concurrent programs. The extensive use of concurrent systems warrants more attention to the concurrent software testing. For testing concurrent software, automatic tools development is getting increased focus. The first part of this study presents a systematic review that aims to explore the state-of-the-art of concurrent software testing. The systematic review reports several issues like concurrent software characteristics, bugs, testing techniques and tools, test case generation techniques and tools, and benchmarks developed for the tools. The second part presents the evaluation of four commercial and open source static analysis tools detecting Java multithreaded bugs. An empirical evaluation of the tools would help the industry as well as the academia to learn more about the effectiveness of the static analysis tools for concurrency bugs.

Context: Software testing is expensive, labor intensive and consumes lot of time in a software development life cycle. There was always a need in software testing to decrease the testing time. This also resulted to focus on Automated Software Testing (AST), because using automated testing, with specific tools, this effort can be dramatically reduced and the costs related with testing can decrease [11]. Manual Testing (MT) requires lot of effort and hard work, if we measure in terms of person per month [11]. Automated Software testing helps to decrease the work load by giving some testing tasks to the computers. Computer systems are cheap, they are faster and don‘t get bored and can work continuously in the weekends. Due to this advantage many researches are working towards the Automation of software testing, which can help to complete the task in less testing time [10]. Objectives: The main aims of this thesis is to 1.) To systematically classify contributions within AST. 2.) To identify the different benefits and challenges of AST. 3.) To identify the whether the reported benefits and challenges found in the literature are prevalent in industry. Methods: To fulfill our aims and objectives, we used Systematic mapping research methodology to systematically classify contributions within AST. We also used SLR to identify the different benefits and challenges of AST. Finally, we performed web based survey to validate the finding of SLR. Results: After performing Systematic mapping, the main aspects within AST include purpose of automation, levels of testing, Technology used, different types of research types used and frequency of AST studies over the time. From Systematic literature review, we found the benefits and challenges of AST. The benefits of AST include higher product quality, less testing time, reliability, increase in confidence, reusability, less human effort, reduction of cost and increase in fault detection. The challenges include failure to achieve expected goals, difficulty in maintenance of test automation, Test automation needs more time to mature, false expectations and lack of skilled people for test automation tools. From web survey, it is observed that almost all the benefits and challenges are prevalent in industry. The benefits such as fault detection and confidence are in contrary to the results of SLR. The challenge about the appropriate test automation strategy has 24 % disagreement from the respondents and 30% uncertainty. The reason is that the automation strategy is totally dependent on the test manager of the project. When asked “Does automated software testing fully replace manual testing”, 80% disagree with this challenge. Conclusion: The classification of the AST studies using systematic mapping gives an overview of the work done in the area of AST and also helps to find research coverage in the area of AST. These results can be used by researchers to use the gaps found in the mapping studies to carry on future work. The results of SLR and web survey clearly show that the practitioners clearly realize the benefits and challenges of AST reported in the literature. / Mobile no: +46723069909

Software testing is becoming more prominent within the automotive industry due to more complex systems, and functions are implemented in the vehicles. The vehicles in the future will have the functionality to manage different levels of automation, which also means that vehicles driven by humans will have more supportive functionality to increase safety and avoid accidents. These functionalities result in a massive growth in the number of test scenarios to indicate that the vehicles are safe, and this makes it impossible to continue performing the tests in the same way as it has been done until today. The new conditions require that the test scenarios and Test Cases both be generated and executed automatically. In this thesis, an investigation and evaluation are performed to analyze the Automatic Test Case Generation methods available for inputs from Natural Language Requirements in an automotive industrial context at NEVS AB. This study aims to evaluate the NAT2TEST strategy by replacing the manual method and obtain a similar or better result. A comparative analysis is performed between the manual and automated approaches for various levels of requirements. The results show that utilizing this strategy in an industrial scenario can improve efficiency if the requirements to be tested are for well-documented lower-level requirements.

This dissertation introduces a hybrid strategy for automated testing of Android applications that combines reinforcement learning and test suite prioritization. These approaches aim to improve the effectiveness of the testing process by employing reinforcement learning algorithms, namely Q-learning and SARSA (State-Action-Reward-State-Action), for automated test case generation. The studies provide compelling evidence that reinforcement learning techniques hold great potential in generating test cases that consistently achieve high code coverage; however, the generated test cases may not always be in the optimal order. In this study, novel test case prioritization methods are developed, leveraging pairwise event interactions coverage, application state coverage, and application activity coverage, so as to optimize the rates of code coverage specifically for SARSA-generated test cases. Additionally, test suite prioritization techniques are introduced based on UI element coverage, test case cost, and test case complexity to further enhance the ordering of SARSA-generated test cases. Empirical investigations demonstrate that applying the proposed test suite prioritization techniques to the test suites generated by the reinforcement learning algorithm SARSA improved the rates of code coverage over original orderings and random orderings of test cases.

Software testing plays a fundamental role in software engineering as it ensures the quality of a software system. However, one of the major challenges of software testing is its costs since it is a time and resource-consuming process which according to academia and industry can take up to 50% of the total development cost. Today, one of the most common ways of generating testcases is through manual labor by analyzing specification documents to produce test scripts, which tends to be an expensive and error prone process. Therefore, optimizing software testing by automating the test case generation process can result in time and cost reductions and also lead to better quality of the end product. Currently, most of the state-of-the-art solutions for automatic test case generation require the usage of formal specifications. Such formal specifications are not always available during the testing process and if available, they require expert knowledge for writing and understanding them. One artifact that is often available in the testing domain is test case specifications written in natural language. In this thesis, an approach for generating integration test cases from natural language test case specifications is designed, applied and, evaluated. Machine learning and natural language processing techniques are used to implement the approach. The proposed approach is conducted and evaluated on an industrial testing project at Ericsson AB in Sweden. Additionally, the approach has been implemented as a tool with a graphical user interface for aiding testers in the process of test case generation. The approach involves performing natural language processing techniques for parsing and analyzing the test case specifications to generate feature vectors that are later mapped to label vectors containing existing C# test scripts filenames. The feature and label vectors are used as input and output, respectively, in a multi-label text classification process. The approach managed to produce test scripts for all test case specifications and obtained a best F1 score of 89% when using LinearSVC as the classifier and performing data augmentation on the training set. / Programvarutestning spelar en grundläggande roll i programvaruutveckling då den säkerställer kvaliteten på ett programvarusystem. En av de största utmaningarna med programvarutestning är dess kostnader eftersom den är en tids och resurskrävande process som enligt akademin och industrin kan ta upp till 50% av den totala utvecklingskostnaden. Ett av de vanligaste sätten att generera testfall idag är med manuellt arbete genom analys av testfallsspecifikationer, vilket tenderar att vara en dyr och felbenägen process. Därför kan optimering av programvarutestning genom automatisering av testfallsgenereringsprocessen resultera i tids- och kostnadsminimeringar och även leda till bättre kvalitet på slutprodukten. Nuförtiden kräver de flesta toppmoderna lösningarna för automatisk testfallsgenerering användning av formella specifikationer. Sådana specifikationer är inte alltid tillgängliga under testprocessen och om de är tillgängliga, så krävs det expertkunskap för att skriva och förstå dem. En artefakt som ofta finns i testdomänen är testfallspecifikationer skrivna på naturligt språk. I denna rapport utformas, tillämpas och utvärderas en metod för generering av integrationstestfall från testfallsspecifikationer skrivna på naturligt språk. Maskininlärnings- och naturlig språkbehandlingstekniker används för implementationen av metoden. Den föreslagna metoden genomförs och utvärderas vid ett industriellt testprojekt hos Ericsson AB i Sverige. Dessutom har metoden implementerats som ett verktyg med ett grafiskt användargränssnitt för att hjälpa testare i testfallsgenereringsprocessen. Metoden fungerar genom att utföra naturlig språkbehandlingstekniker på testfallsspecifikationer för att generera egenskapsvektorer som senare mappas till etikettsvektorer som innehåller befintliga C# testskriptfilnamn. Engenskaps och etikettsvektorerna används sedan som indata och utdata, respektive, för textklassificeringsprocessen. Metoden lyckades producera testskript för allatestfallsspecifikationer och fick en bästa F1 poäng på 89% när LinearSVC användes för klassificeringen och datautökning var utförd på träningsdatat.

The software industry is expanding at a rapid rate. To keep up with the fast-growing and ever-changing technologies, it has become necessary to produce high-quality software in a short time and at an affordable cost. This research aims to demonstrate to Java developers the use of Automated Test Case Generation (ATCG) tools by presenting the tools that exist today, their usefulness, and their effectiveness. The main focus is on the automated testing tools for the Java industry, which can help developers achieve their goals faster and make better software. Moreover, the discussion covers the availability, features, prerequisites, effectiveness, and limitations of the automated testing tools. Among these tools, the most widely used are Evosuite, JUnit, TestNG, and Selenium. Each tool has its advantages and purpose. Furthermore, these ATCG-tools were compared to provide a clear picture to Java developers, answer the research questions, and show strengths and limitations of each selected tool. Results show that there is no single ultimate tool that can do all kinds of testing independently. It all depends on what the developer aims to achieve. If one tool is good at generating unit test cases for Java classes, another tool is good at testing the code security through penetration testing. Therefore, the Java developers may choose a tool/s based on their requirements. This study has revealed captivating findings regarding the ATCG-tools, which ought to be explored in the future.