Regression testing is the process of testing software to make sure changes to the software will not change the functionality. With growing test suites theneed to prioritize arises. This thesis explores how to weigh factors such as the number of fails detected, days since latest test case execution, and coverage. The prioritization is done over multiple test systems, software branches, and over many test sessions where the software can change in-between. With data provided by an industrial partner, we evaluate different ways to prioritize. The developed mathematical model could not cope with the size of the problem, whereas a simulated annealing approach based on said model proved highly successful. We also found that prioritizing test cases related to recent codechanges was effective. / Regressionstestning är processen att testa mjukvara för att säkerställa att ändringar av mjukvaran inte kommer att ändra funktionaliteten. Med växande testsviter uppstår behovet av att prioritera. Det här examensarbetet undersöker hur man väger faktorer som antalet upptäckta underkända testfall, dagar sedan testfallen senast kördes och täckning. Prioriteringen görs över flera testsystem, mjukvarugrenar och över många testsessioner där mjukvaran kan ändras däremellan. Med data från en industriell partner utvärderar vi olika sätt att prioritera. Den utvecklade matematiska modellen kunde inte hantera problemets storlek, medan en simulerad kylningsmetod baserad på denna modell visade sig vara mycket framgångsrik. Vi fann också att prioritering enligt ändringar som gjorts i mjukvaran var effetivt

Software testing is a very important phase in software development as it minimize risks ina software system, however, it consumes time and can be very expensive. With automatictest case generation time consumption and cost can be reduced. Model-based testing isa method to test a software system with a model of the systems behaviour. Automatictest case generation is often considered a favorable support in model-based testing. In thiswork, the concept of model-based testing is explored along with testing the embedded partof a water-level measurement system (WLM) to investigate the efficiency of model-basedtesting on a software system. As a result of this, a model-based testing tool, MoMut::UMLis used to generate the test-cases on the UML model of WLM system that is built ina UML modeling environment, Eclipse-Papyrus. However, MoMut::UML implements aspecial type of model-based testing technique, model-based mutation testing; that injectsfaults in the UML model, and generates test-data on the fault-based model. By this, thebehaviour of system-under-test, only the UML model of water-level measurement system,is tested.

Unit testing plays a crucial role in ensuring the quality and reliability of software systems. However, manual testing can often be a slow and time-consuming process. With current advancements in artificial intelligence (AI), new tools have emerged for automated unit testing to address this issue. But how do these new AI tools compare to conventional automated unit test generation tools? To answer this question, we compared two state-of-the-art conventional unit test tools (EVOSUITE and RANDOOP) with the sole commercially available AI-based unit test tool (DIFFBLUE COVER) for Java. We tested them on 10 sample classes from 3 real-life projects provided by the Defects4J dataset to evaluate their performance regarding code coverage, mutation score, and fault detection. The results showed that EVOSUITE achieved the highest code coverage, averaging 89%, while RANDOOP and DIFFBLUE COVER achieved similar results, averaging 63%. In terms of mutation score, DIFFBLUE COVER had the lowest average score of 40%, while EVOSUITE and RANDOOP scored 67% and 50%, respectively. For fault detection, EVOSUITE and RANDOOP detected a higher number of bugs (7 out of 10 and 5 out of 10, respectively) compared to DIFFBLUE COVER, which found only 4 out of 10. Although the AI-based tool was outperformed in all three criteria, it still shows promise by being able to achieve adequate results, in some cases even surpassing the conventional tools while generating a significantly smaller number of total assertions and more comprehensive tests. Nonetheless, the study acknowledges its limitations in terms of the restricted number of AI-based tools used and the small number of projects utilized from Defects4J.

As the field of computer science continues to grow, the number of students enrolled in related programs will grow as well. Though one-on-one tutoring is one of the more effective means of teaching, computer science instructors will have less and less time to devote to individual students. To address this growing concern, many tools that automate parts of an instructor’s job have been proposed. These tools can assist instructors in presenting concepts and grading student work, and they can help students learn to program more effectively. A growing group of intelligent tutoring systems attempts to tie all of this functionality into a single tool that is meant to be used throughout an entire CS course or series of courses. To contribute to this emerging area, the Factoring Problem Generator (FPG) is presented in this work. The FPG creates and grades problems in C in which students search for and extract blocks of repeated code into individual functions, learning to utilize parameters and return values as they do so. The problems created by the FPG are highly configurable by instructors such that the difficulty can be finely tuned to suit students’ individual needs. Instructors can choose whether or not to include arrays, pointers, certain elemental data types, certain operators, or certain kinds of statements, among other things. The FPG is additionally capable of generating a set of test cases for each generated problem. These test cases fully exercise students’ solutions by covering all branches of execution, and they ensure that program functionality does not change as students factor code into functions. Initial experimentation with the system has suggested that the FPG can be integrated into a beginning CS curriculum and with further refinement could become a standard tool in the CS classroom.

At present, there are different Electronic Identity (eID) systems utilized in EU, resulting in difficulties to carry eID information and transfer eID data of e-learning systems from one EU country to another. A project entitled Secure idenTity acrOss boRders linKed (STORK) was launched to address this problem, by installing a Pan European Proxy Service (PEPS) server. Currently, Logica, a Swedish company, cooperates with Department of Computer and Systems Sciences (DSV) to implement the PEPS at DSV. This thesis aims to build various testing cases for the PEPS server installed at DSV. The PEPS is well-developed and separate packages working together with the Service Provider (SP) and Identity Provider (IDP) to implement its respective functionalities. The tests performed on PEPS are used to test the whole PEPS infrastructure: SP, PEPS and IDP, that is, the communication between these packages. The purpose of implementation of PEPS is to support the Ilearn@DSV to connect with STORK. Hence, the SP in this thesis is Ilearn@DSV system embedded with SP package. This thesis first introduces the background of eID, e-learning and the e-learning system Ilearn@DSV. Then, it describes the test hierarchy and test requirements, and completes data collection step. The details of various test cases are provided for the predetermined test items in test plans. Test plans and test cases must abide by the IEEE test format and meet the IEEE Standard 829-2008. Finally, test cases are validated against depth, breadth and effectiveness. / För närvarande finns det olika elektroniska identitetssystem (eID) som används inom EU, vilket resulterar i svårigheter att använda eID-information och överföra e-lärande systems eID-data från ett EU-land till annat. Projektet Secure idenTity acrOss boRders linKed (STORK) lanserades för att lösa detta problem, genom att installera en Pan European Proxy Service (PEPS)-server. Nu samarbetar Logica, ett svenskt företag, med Institutionen för data- och systemvetenskap (DSV) för att implementera PEPS på DSV. Detta examensarbete syftar till att bygga olika testfall för PEPS-servern installerad hos DSV. PEPS är välutvecklat och separata paket som arbetar tillsammans med Service Provider (SP) och Identity Provider (IDP) för att implementera respektiv funktioner. Testerna som utförs på PEPS används för att testa hela PEPS-infrastrukturen: SP, PEPS och IDP, det vill säga kommunikationen mellan dessa paket. Syftet med implementeringen av PEPS är att stödja Ilearn@DSV för att få kontakt med STORK. Därför är SP i detta examensarbete Ilearn@DSV-systemet inbyggt i SP-paket. Detta examensarbete först introducerar bakgrunden till eID, e-learning och e-learning-systemet Ilearn@DSV. Sedan beskriver den testhierarkin och testkraven och slutför datainsamlingssteget. Detaljerna för olika testfall tillhandahålls för de förutbestämda testobjekten i testplanerna. Testplaner och testfall får följa IEEE-testformatet och uppfylla IEEE Standard 829-2008. Slutligen valideras testfall mot djup, bredd och effektivitet.

Context: Testing is an important activity in software development and is typically estimated to account for nearly half of the efforts in the software development cycle. This puts a great demand on improving the artifacts involved in this task such as the test cases and test suites (a collection of test cases).  Objective: When executing test programs, it is typical to record runtime information associated with the test cases in the form of test execution logs or traces. The aim of this work is to explore how this information can be utilized to improve the software testing process. To this end, two main aspects are investigated which are (1) in the context of test case generation and (2) in the context of different optimizations regarding existing test suites. Furthermore, the role of the logs regarding fault localization in connection with improving the existing test suites is investigated. Method: A systematic literature review is conducted to investigate, identify and analyze the existing literature on test case generation and test suite optimization that utilizes the test execution logs. Results: After a rigorous search in six digital databases, 26 primary studies were identified. 5 of the selected papers propose approaches in the context of test data generation, 8 papers suggest test case prioritization (TCP) techniques, 4 papers discuss approaches in test case selection (TCS), and 5 papers propose approaches in test suite minimization (TSM). Furthermore, we identified, 3 papers that discuss fault localization, and one paper that discussed the decomposition of large test cases into smaller single purpose test cases using the logs from previous test executions. Conclusion: The test execution logs are a useful source of information for different testing activities. Regarding test case generation, the main theme observed is the use of genetic algorithms in attempting to generate appropriate test cases when the alternative might have been to use random test data generation methods. When it comes to improving existing test suites several approaches within TCP, TCS and TSM such as similarity-based, modification-based, cluster-based, and search-based were put forward by the authors of the selected primary studies. Furthermore, several fault localization techniques using the logs were suggested.

This study investigates the utilization of Large Language Models for Test Case Generation. The study uses the Large Language model and Embedding model provided by Llama, specifically Llama2 of size 7B, to generate test cases given a defined input. The study involves an implementation that uses customization techniques called Retrieval Augmented Generation (RAG) and Prompt Engineering. RAG is a method that in this study, stores organisation information locally, which is used to create test cases. This stored data is used as complementary data apart from the pre-trained data that the large language model has already trained on. By using this method, the implementation can gather specific organisation data and therefore have a greater understanding of the required domains. The objective of the study is to investigate how AI-driven test case generation impacts the overall software quality and development efficiency. This is evaluated by comparing the output of the AI-based system, to manually created test cases, as this is the company standard at the time of the study. The AI-driven test cases are analyzed mainly in the form of coverage and time, meaning that we compare to which degree the AI system can generate test cases compared to the manually created test case. Likewise, time is taken into consideration to understand how the development efficiency is affected. The results reveal that by using Retrieval Augmented Generationin combination with Prompt Engineering, the system is able to identify test cases to a certain degree. The results show that 66.67% of a specific project was identified using the AI, however, minor noise could appear and results might differ depending on the project’s complexity. Overall the results revealed how the system can positively impact the development efficiency and could also be argued to have a positive effect on the software quality. However, it is important to understand that the implementation as its current stage, is not sufficient enough to be used independently, but should rather be used as a tool to more efficiently create test cases.

Despite much scepticism and problems for its adoption, the Model-Driven Development (MDD) is being used and improved to provide many inherent benefits for industry. One of its greatest benefits is the ability to handle the complexity of software development by raising the abstraction level. Models are expressed using concepts that are not related to a specific implementation technology (e.g. Unified Modelling Language -UML, Object Constraint Language -OCL, Action Language for Foundational UML -ALF), which means that the models can be easier to specify, maintain and document. As in Model-Driven Engineering (MDE), the primary artefacts are the conceptual models, efforts are focused on their creation, testing and evolution at different levels of abstraction through transformations because if a conceptual schema has defects, these are passed on to the following stages, including coding. Thus, one of the challenges for researchers and developers in Model-Driven Development is being able to identify defects early on, at the conceptual schema level, as this helps reduce development costs and improve software quality. Over the last decade, little research work has been performed in this area. Some of the causes of this are the high theoretical complexity of testing conceptual schemas and the lack of adequate software support. This research area thus admits new methods and techniques, facing challenges such as generation of test cases using information external to the conceptual schemas (i.e. requirements), the measurement of possible automation, selection and prioritization of test cases, the need for an efficient support tool using standard semantics, the opportune feedback to support the software quality assurance process and facilitate making decisions based on the analysis and interpretation of the results. The aim of this thesis is to mitigate some of the problems that affect conceptual schema validation by providing a novel testing-based validation framework based on Model-Driven Development. The use of MDD improves abstraction, automation and reuse, which allows us to alleviate the complexity of our validation framework. Furthermore, by leveraging MDD techniques (such as metamodeling, model transformations, and models at runtime), our framework supports four phases of the testing process: test design, test case generation, test case execution and the evaluation of the results. In order to provide software support for our proposal, we developed the CoSTest ALF-based testing environment. To ensure that CoSTest offers the necessary functionality, we first identified a set of functional requirements. Then, after these requirements were identified, we defined the architecture and testing environment of the validation framework, and finally we implemented the architecture in the Eclipse context. CoSTest has been developed to test several properties on the executable model, such as syntactic correctness (i.e. all the elements in the model conform to the syntax of the language in which it is described), consistency between the structural and behavioural parts (its integrity constraints) and completeness (i.e. all possible changes on the system state can be performed through the execution of the operations defined in the executable model). For defective models, the CoSTest report returns a meaningful feedback that helps locate and repair any defects detected. / A pesar del escepticismo y dificultades en su adopción, el Desarrollo Orientado por Modelos (MDD, por sus siglas en inglés) está siendo usado y mejorado para proveer muchos beneficios inherentes a la industria. Uno de sus mayores beneficios es la capacidad de manejar la complejidad del desarrollo de software elevando el nivel de abstracción. Los modelos se expresan utilizando conceptos que no están relacionados con una tecnología de implementación específica (por ejemplo, Lenguaje de Modelado Unificado -UML, Lenguaje de Restricción de Objetos -OCL, Lenguaje de Acción para el Foundational UML - ALF), lo que significa que los modelos pueden ser más fáciles de especificar, mantener y documentar. Debido a que en una Ingeniería dirigida por modelos (MDE), los artefactos primarios son los modelos conceptuales, los esfuerzos se centran en su creación, prueba y evolución a diferentes niveles de abstracción a través de transformaciones, porque si un esquema conceptual tiene defectos, éstos se pasan a las siguientes etapas, incluida la codificación. Por lo tanto, uno de los retos para los investigadores y desarrolladores in MDD es poder identificar los defectos temprano, a nivel de esquemas conceptuales, ya que esto ayudaría a reducir los costos de desarrollo y mejorar la calidad del software. Durante la última década, pocos trabajos de investigación se han realizado en esta área. Algunas de las causas de esta realidad son la alta complejidad teórica de probar esquemas conceptuales y la falta de soporte de software adecuado. Por lo tanto, este área de investigación admite nuevos métodos y técnicas, enfrentando retos como la generación de casos de prueba utilizando información externa a los esquemas conceptuales (es decir, los requisitos), la medición de una posible automatización, selección y priorización de casos de prueba, la necesidad de una herramienta de soporte eficiente que utilice una semántica estándar, la retroalimentación oportuna para apoyar el proceso de aseguramiento de la calidad del software y facilitar la toma de decisiones basadas en el análisis y la interpretación de los resultados. El objetivo de esta tesis es mitigar algunos de los problemas que afectan la validación de los esquemas conceptuales, proporcionando un nuevo marco de validación basado en pruebas que fue construido usando un desarrollo dirigido por modelos. El uso de MDD permite un aumento en la abstracción, automatización y reutilización que nos permite aliviar la complejidad de nuestro marco de validación. Además, al aprovechar las técnicas MDD (como el metamodelado, las transformaciones de modelos y los modelos en tiempo de ejecución), nuestro marco soporta cuatro fases del proceso de prueba: diseño de pruebas, generación de casos de prueba, ejecución de casos de prueba y la evaluación de los resultados. Con el fin de proporcionar soporte de software para nuestra propuesta, hemos desarrollado CoSTest, un entorno de pruebas basado en el lenguaje ALF. Para asegurar que CoSTest ofrece la funcionalidad necesaria, primero identificamos un conjunto de requisitos funcionales. Luego, después de identificar estos requisitos, definimos la arquitectura y el ambiente de pruebas de nuestro marco de validación y, finalmente, implementamos la arquitectura en el contexto de Eclipse. CoSTest ha sido desarrollado para probar varias propiedades sobre el modelo ejecutable como la corrección sintáctica (es decir, todos los elementos del modelo se ajustan a la sintaxis del lenguaje en el que se describe), consistencia entre la parte estructural y el comportamiento (sus restricciones de integridad) y completitud (es decir, todos los cambios posibles en el estado del sistema se pueden realizar a través de la ejecución de las operaciones definidas en el modelo ejecutable). Para los modelos defectuosos, el informe de CoSTest devuelve una retroalimentación significativa que ayuda a localizar y reparar los defectos detec / A pesar de l'escepticisme i les dificultats en la seua adopció, el Desenvolupament Orientat per Models (MDD, segons les sigles en anglès) està sent usat i millorat per tal de proveir molts beneficis potencials inherents a l' indústria. Un dels majors beneficis és la capacitat de manejar la complexitat del desenvolupament del programari elevant el nivell d'abstracció. Els models s'expressen mitjançant conceptes que no estan relacionats amb una tecnologia d'implementació específica (per exemple, el Llenguatge de Modelat Unificat - UML, Llenguatge de Restricció d'Objectes -OCL, Llenguatge d'Acció per al Foundational UML - ALF), el que significa que els models poder ser més fàcils d'especificar, mantindre i documentar. A causa de que en una Enginyeria dirigida per models (MDE), els artefactes primaris són els models conceptuals, els esforços es centren en la seua creació, prova i evolució a diferents nivells d'abstracció mitjançant transformacions, perquè si un esquema conceptual té defectes, aquestos es passen a les següents etapes, inclosa la codificació. Per tant, un del reptes per als investigadors i desenvolupadors en MDD és poder identificar els defectes des del principi, a nivell de esquemes conceptuals, perquè açò ajudaria a reduir els costos de desenvolupament i millora de la qualitat del programari. Durant l'última dècada, pocs treballs d'investigació s'han fet en aquesta àrea. Algunes de les causes d'aquesta realitat són l'alta complexitat teòrica de provar esquemes conceptuals i la falta de suport de programari adequat. Per tant, aquesta àrea d'investigació admet nous mètodes i tècniques, enfrontant reptes com la generació de casos de prova mitjançant informació externa als esquemes conceptuals (es a dir, requisits), la medició de una possible automatització, selecció i priorització de casos de prova, la necessitat de una ferramenta de suport rentable que utilitze una semàntica estàndard, la retroalimentació oportuna per suportar el procés d'assegurament de la qualitat del programari i la facilitat per a prendre decisions basades en l'anàlisi i la interpretació dels resultats. En aquesta tesi intentem mitigar alguns dels problemes que afecten a la validació dels esquemes conceptuals, proporcionant un nou marc de validació basat en proves que va ser construït mitjançant un desenvolupament dirigit per models. L'ús de MDD permet un augment en l'abstracció, automatització i reutilització que ens permet alleujar la complexitat del nostre marc de validació. A més a més, al aprofitar les tècniques MDD (com el metamodelat, les transformacions de models i els models en temps d'execució), el nostre marc suporta quatre fases del procés de prova: disseny, generació i execució de casos de prova, així com l'avaluació de resultats del procés de prova. Amb la finalitat de proporcionar suport de programari per a la nostra proposta, hem desenvolupat un entorn de proves basat en el llenguatge ALF que s'anomena CoSTest. Per tal d'assegurar que CoSTest ofereix la funcionalitat necessària, identifiquem un conjunt de requisits funcionals abans de desenvolupar la ferramenta. Després d'identificar aquestos requisits, definim l'arquitectura i l'ambient de proves del nostre marc de validació, i finalment, implementem l'arquitectura en el context Eclipse. CoSTest ha sigut desenvolupat per provar diverses propietats sobre el model executable com la correcció sintàctica (és a dir, tots els elements del model s'ajusten a la sintaxi del llenguatge en el que es descriu), consistència antre la part estructural i el comportament (les seues restriccions d'integritat) i completitud (és a dir, tots els canvis possibles en l'estat del sistema es poden realitzar mitjançant l'execució de les operacions definides en el model executable). Per als models defectuosos, l'informe de CoSTest retorna una retroalimentació significativa que ajuda a localitzar i reparar els defectes dete / Granda Juca, MF. (2017). Testing-Based Conceptual Schema Validation in a Model-Driven Environment [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/89091

Thesis (MSc)--Stellenbosch University, 2013. / ENGLISH ABSTRACT: In this dissertation we present a software tool, Artemis, that symbolically executes Java virtual machine bytecode to find bugs and automatically generate test cases to trigger the bugs found. Symbolic execution is a technique of static software analysis that entails analysing code over symbolic inputs—essentially, classes of inputs—where each class is formulated as constraints over some input domain. The analysis then proceeds in a path-sensitive way adding the constraints resulting from a symbolic choice at a program branch to a path condition, and branching non-deterministically over the path condition. When a possible error state is reached, the path condition can be solved, and if soluble, value assignments retrieved to be used to generate explicit test cases in a unit testing framework. This last step enhances confidence that bugs are real, because testing is forced through normal language semantics, which could prevent certain states from being reached. We illustrate and evaluate Artemis on a number of examples with known errors, as well as on a large, complex code base. A preliminary version of this work was successfully presented at the SAICSIT conference held on 1–3 October 2012, in Centurion, South Africa. / AFRIKAANSE OPSOMMING: In die dissertasie bied ons ’n stuk sagtewaregereedskap, Artemis, aan wat biskode van die Java virtuele masjien simbolies uitvoer om foute op te spoor en toetsgevalle outomaties voort te bring om die foute te ontketen. Simboliese uitvoering is ’n tegniek van statiese sagteware-analise wat behels dat kode oor simboliese toevoere—in wese, klasse van toevoer—geanaliseer word, waar elke klas geformuleer word as beperkinge oor ’n domein. Die analise volg dan ’n pad-sensitiewe benadering deur die domeinbeperkinge, wat volg uit ’n simboliese keuse by ’n programvertakking, tot ’n padvoorwaarde by te voeg en dan nie-deterministies vertakkings oor die padvoorwaarde te volg. Wanneer ’n moontlike fouttoestand bereik word, kan die padvoorwaarde opgelos word, en indien dit oplaasbaar is, kan waardetoekennings verkry word om eksplisiete toetsgevalle in ’n eenheidstoetsingsraamwerk te formuleer. Die laaste stap verhoog vertroue dat die foute gevind werklik is, want toetsing word deur die normale semantiek van die taal geforseer, wat sekere toestande onbereikbaar maak. Ons illustreer en evalueer Artemis met ’n aantal voorbeelde waar die foute bekend is, asook op ’n groot, komplekse versameling kode. ’n Voorlopige weergawe van die´ werk is suksesvol by die SAICSIT-konferensie, wat van 1 tot 3 Oktober 2012 in Centurion, Suid-Afrika, gehou is, aangebied.

Regression testing is just as important for the quality assurance of a system, as it is time consuming. Several techniques exist with the purpose of lowering the execution times of test suites and provide faster feedback to the developers, examples are ones based on transition-models or string-distances. These techniques are called test case selection (TCS) techniques, and focuses on selecting subsets of the test suite deemed relevant for the modifications made to the system under test. This thesis project focused on evaluating the use of a topic model, latent dirichlet allocation, as a means to create a diverse selection of test cases for coverage of certain test characteristics. The model was tested on authentic data sets from two different companies, where the results were compared against prior work where TCS was performed using similarity-based techniques. Also, the model was tuned and evaluated, using an algorithm based on differential evolution, to increase the model’s stability in terms of inferred topics and topic diversity. The results indicate that the use of the model for test case selection purposes was not as efficient as the other similarity-based selection techniques studied in work prior to thist hesis. In fact, the results show that the selection generated using the model performs similar, in terms of coverage, to a randomly selected subset of the test suite. Tuning of the model does not improve these results, in fact the tuned model performs worse than the other methods in most cases. However, the tuning process results in the model being more stable in terms of inferred latent topics and topic diversity. The performance of the model is believed to be strongly dependent on the characteristics of the underlying data used to train the model, putting emphasis on word frequencies and the overall sizes of the training documents, and implying that this would affect the words’ relevance scoring to the better.