Traditional test oracles have two problems. Firstly, several test oracles are needed for a single software program to perform different functions and maintaining a large number of test oracles is tedious and might be prone to errors. Secondly, testers usually test only the important criteria of a web application, since its time consuming to check with all the possible criteria. Ontologies have been used in a wide variety of domains and they have also been used in software testing. However, they have not been used for test oracle automation. The main idea of this thesis is to define a procedure for how ontology-based test oracle automation can be achieved for testing web applications and minimize the problems of traditional test oracles. The proposed procedure consists of the following steps: first, the expected results are stored in ontology A by running previous working version of the web application; second, the actual results are stored in ontology B by running the web application under test at runtime; and finally, the results of both ontology A and B are compared. This results in an automated test oracle comparator. Evaluation includes how the proposed procedure minimizes the traditional test oracle problems and by identifying the benefits of the defined procedure.

Financial markets use complex computer trading systems whose failures can cause serious economic damage, making reliability a major concern. Automated random testing has been shown to be useful in finding defects in these systems, but its inherent test oracle problem (automatic generation of the expected system output) is a drawback that has typically prevented its application on a larger scale. Two main tasks have been carried out in this thesis as a solution to the test oracle problem. First, an independent model of a real trading system based on constraint programming, a method for solving combinatorial problems, has been created. Then, the model has been integrated as a true test oracle in automated random tests. The test oracle maintains the expected state of an order book throughout a sequence of random trade order actions, and provides the expected output of every auction triggered in the order book by generating a corresponding constraint program that is solved with the aid of a constraint programming system. Constraint programming has allowed the development of an inexpensive, yet reliable test oracle. In 500 random test cases, the test oracle has detected two system failures. These failures correspond to defects that had been present for several years without being discovered neither by less complete oracles nor by the application of more systematic testing approaches. The main contributions of this thesis are: (1) empirical evidence of both the suitability of applying constraint programming to solve the test oracle problem and the effectiveness of true test oracles in random testing, and (2) a first attempt, as far as the author is aware, to model a non-theoretical continuous double auction using constraint programming. / Winner of the Swedish AI Society's prize for the best AI Master's Thesis 2010.

Embedded systems are present in many fields of application where failure may be critical. Such systems often possess characteristics that hampers the testing activity, as large amount of produced data and temporal requirements which must be specified and evaluated. There are tools that support the development of models for analysis and simulation still in the design stage. After being evaluated, a model may be used as basis to the implementation. In this case, it is important to ensure that the model is consistent with the specification. Otherwise, a divergence will be propagated to the final code. Therefore, the model must be tested prior to the codification. Simulink is a standard development and simulation tool for models of embedded systems. Its wide application in the industry has promoted the creation of free-software alternatives, as XCos. In the literature, there are researches which seek to improve the testing activity for Simulink-like models. The proposed solutions usually focus on test case selection strategies. However, little efforts have been directed to the oracle problem, that is, the difficulty in evaluating if an execution agrees with the specification. The objective of this doctorate proposal is to provide an oracle generation approach for Simulink-like models which addresses the characteristics previously summarized. Specifically, it is proposed a process, methods, procedures and a tool that enable the partially-automated generation of oracles for such models. As a main contribution, it is expected an improvement in the evaluation process of embedded systems in terms of quality, cost and time / Sistemas embarcados estão presentes em diversas áreas de aplicação em que falhas podem ser críticas. Tais sistemas frequentemente possuem características que tornam a fase de teste particularmente desafiadora, como a produção de grande quantidade de dados e requisitos temporais que precisam ser validados de acordo com a especificação. Existem ferramentas que auxiliam no desenvolvimento de modelos para análise e simulação do comportamento de sistemas embarcados ainda na fase de design. Após ser avaliado, o modelo pode ser usado como base para a implementação. Neste caso, deve-se buscar garantir que um modelo esteja de acordo com a especificação. Do contrário, tal divergência será propagada para a implementação. Portanto, e importante que o modelo seja testado antes da fase de implementação. Simulink e uma ferramenta-padrão de desenvolvimento e simulação de modelos de sistemas embarcados. Sua ampla aplicação na indústria incentivou a criação de alternativas de software livres como XCos. Na literatura, existem pesquisas que visam a aprimorar a atividade de teste de modelos Simulink-like. As soluções propostas geralmente focam em estratégias de seleção de casos de teste. Mas pouco esforço tem sido direcionado ao problema do oráculo, isto e, na dificuldade em avaliar se a execução está de acordo com a especificação. O objetivo desta proposta de doutorado é prover uma abordagem de geração de oráculos de teste para modelos simulink-like que contemple as características previamente resumidas. Especificamente, é proposto um processo, métodos, procedimentos e uma ferramenta que viabilizem a geração parcialmente automatizada de oráculos de teste para modelos Simulink-like. Como contribuição principal, é esperada a melhora da qualidade, custo e tempo do processo de validação de sistemas embarcados suportados por modelagem em Simulink e ferramentas similares

Kasdieninis žiniatinklio informacinių sistemų (toliau – IS) naudojimas paprastų vartotojų, verslo ir vyriausybinių organizacijų tarpe, reikalauja vis aukštesnės tokių IS kokybės. Žiniatinklio IS dažniausiai yra tokia programinė įranga, kurios sudėtingumas, dėl keleto apjungiamų technologijų, yra aukštas, o pokyčių greitis yra didelis. Vykdant palaikymo ir priežiūros darbus, gali atsirasti begalė klaidų, kurias iššaukia atlikti pakeitimai. Regresinio testavimo paskirtis yra užtikrinti, jog įdiegus programinės įrangos pakeitimus, tokius kaip naujo funkcionalumo pridėjimas ar jau egzistuojančio taisymas, ankstesnėje IS versijoje veikęs funkcionalumas veiks taip pat gerai ir naujojoje versijoje. Šiame darbe tiriami iš HTML žymų sudarytą IS atsaką galintys palyginti regresinio testavimo algoritmai (toliau – testų orakulai). HTML ãtsako kaip teksto lyginimas pateikia per daug klaidingai teigiamų (angl. false positives) rezultatų – testuotojas užverčiamas pranešimais apie netikras klaidas. Mūsų realizuotas testų orakulas HTML žymų atsaką nagrinėja kaip medžio struktūrą. Toks algoritmo veikimas leidžia tikrinti arba netikrinti aprašytas HTML semantines savybes, ko pasekoje yra sumažinamas klaidingai teigiamų ir klaidingai neigiamų rezultatų kiekis. Realizuotas testų orakulas nuo literatūroje aprašomų ir rinkoje siūlomų testų orakulų skiriasi tuo, jog nepraneša apie klaidą, kai HTML atsakas būna papildytas naujomis žymomis, kurios nekeičia pradinės medžio struktūros. Žiniatinklio IS... [toliau žr. visą tekstą] / Common usage of web information systems among consumers, business and governmental organizations require higher quality of such systems. Web information systems is such a software, which has high complexity because of the multiple technology usage and tends to rapid changes. Errors may occur because of the changes that have been made through maintenance period. Regression testing purpose is to ensure, that software changes such as new functionality or previously existed functionality modification, did not affect functionality, which was well working in the older version. In these master thesis we analyze test oracle comparators which can compare HTML response. HTML as a text comparison produces too many false positives. Meanwhile our test oracle comparator analyses HTML response as is was a tree structure. Thus implementation of the test oracle comparator algorithm enables opportunity to validate or not to validate described HTML semantic features. Therefore, we get fewer false positives and false negatives. Our implemented test oracle comparator differs from the ones described in literature or the ones proposed in market by capability not to warn tester about changes such as newly added HTML tag, which do not change overall tree structure. In this way the introduction of natural functionality growth is allowed. After improvement our test oracle comparator detects 90 % errors with the 90 % precision, whereas in the market existing test oracle comparator of the regression... [to full text]

Embedded systems are present in many fields of application where failure may be critical. Such systems often possess characteristics that hampers the testing activity, as large amount of produced data and temporal requirements which must be specified and evaluated. There are tools that support the development of models for analysis and simulation still in the design stage. After being evaluated, a model may be used as basis to the implementation. In this case, it is important to ensure that the model is consistent with the specification. Otherwise, a divergence will be propagated to the final code. Therefore, the model must be tested prior to the codification. Simulink is a standard development and simulation tool for models of embedded systems. Its wide application in the industry has promoted the creation of free-software alternatives, as XCos. In the literature, there are researches which seek to improve the testing activity for Simulink-like models. The proposed solutions usually focus on test case selection strategies. However, little efforts have been directed to the oracle problem, that is, the difficulty in evaluating if an execution agrees with the specification. The objective of this doctorate proposal is to provide an oracle generation approach for Simulink-like models which addresses the characteristics previously summarized. Specifically, it is proposed a process, methods, procedures and a tool that enable the partially-automated generation of oracles for such models. As a main contribution, it is expected an improvement in the evaluation process of embedded systems in terms of quality, cost and time / Sistemas embarcados estão presentes em diversas áreas de aplicação em que falhas podem ser críticas. Tais sistemas frequentemente possuem características que tornam a fase de teste particularmente desafiadora, como a produção de grande quantidade de dados e requisitos temporais que precisam ser validados de acordo com a especificação. Existem ferramentas que auxiliam no desenvolvimento de modelos para análise e simulação do comportamento de sistemas embarcados ainda na fase de design. Após ser avaliado, o modelo pode ser usado como base para a implementação. Neste caso, deve-se buscar garantir que um modelo esteja de acordo com a especificação. Do contrário, tal divergência será propagada para a implementação. Portanto, e importante que o modelo seja testado antes da fase de implementação. Simulink e uma ferramenta-padrão de desenvolvimento e simulação de modelos de sistemas embarcados. Sua ampla aplicação na indústria incentivou a criação de alternativas de software livres como XCos. Na literatura, existem pesquisas que visam a aprimorar a atividade de teste de modelos Simulink-like. As soluções propostas geralmente focam em estratégias de seleção de casos de teste. Mas pouco esforço tem sido direcionado ao problema do oráculo, isto e, na dificuldade em avaliar se a execução está de acordo com a especificação. O objetivo desta proposta de doutorado é prover uma abordagem de geração de oráculos de teste para modelos simulink-like que contemple as características previamente resumidas. Especificamente, é proposto um processo, métodos, procedimentos e uma ferramenta que viabilizem a geração parcialmente automatizada de oráculos de teste para modelos Simulink-like. Como contribuição principal, é esperada a melhora da qualidade, custo e tempo do processo de validação de sistemas embarcados suportados por modelagem em Simulink e ferramentas similares

The train represents a complex system, where every sub-system has an important role. If a subsystem doesn’t work how it should, the correctness of whole the train can be uncertain. To ensure that system works properly, we should test each sub-system individually and integrate them together in the whole system. Each of these subsystems consists of the different modules with different functionalities what should be tested. Testing of different functionalities often requires a different approach. For some functionalities, it is necessary domain knowledge from the human expert, such as classification of signals in different use cases in Propulsion and Controls (PPC) in Bombardier Transportation. Due to this reason, we need to simulate of using experts knowledge in the certain domain. We are investigating the use of machine learning techniques for solving this cases and creating system what will automatically classify different signals using the previous human knowledge. This case study is conducted in Bombardier Transportation (BT), Västerås in departments Train Control Management System (TCMS) and Propulsion and Controls (PPC), where data is collected, analyzed and evaluated. We proposed a method for solving the oracle problem based on machine learning approach for different for certain use case. Also, we explained different steps what can be used for solving the test oracle problem where signals are part of verdict process

Model-Based Testing has shown tangible benefits in previous research exploring widespread domains. Clearing systems in financial markets however are large software systems in which these approaches rarely have been explored. This paper in particular explores the Model-Based Testing approach using Modbat, a specialized tool based on extended finite-state machines, as a complementary testing approach on software clearing systems at Nasdaq. In particular, the approach is applied to functionalities of the system to test both its ability to find bugs undiscovered by traditional methods, as well as to measure its effectiveness and efficiency in creating reliable coverage in short amounts of time. In this study, we show that the model-based Modbat approach can achieve high code coverage with relatively little effort. 77 % of the production coverage was achieved with very few Modbat test runs. Additionally, only ∼ 9-14 % of the same lines of code, and a fraction of the time were required to achieve these results. From this, we conclude that Model-Based Testing approaches such as Modbat allow for significantly more efficient testing in terms of coverage over time compared to manual approaches. / Modellbaserad testning har visat på konkreta fördelar i tidigare forskning som utforskat omfattande områden. Clearingsystem inom finansmarknader är dock stora programvarusystem där dessa metoder sällan har utforskats. Denna arbete utforskar specifikt modellbaserad testning med användning av Modbat, ett specialiserat verktyg baserat på utökade ändliga automater, som ett kompletterande testningsmetodik för programvarurensningssystem på Nasdaq. Särskilt tillämpas metoden på specifika funktioner i systemet för att testa både dess förmåga att hitta buggar som inte upptäckts av traditionella metoder och att mäta dess effektivitet att skapa pålitlig täckning av kod på kort tid. I denna studie visar jag att den modellbaserade Modbat-metoden kan uppnå hög kodtäckning med relativt lite ansträngning. 77 % av produktionskoden täcktes med mycket få Modbat-testkörningar. Dessutom krävdes endast cirka 9-14 % av samma LOC, och en bråkdel av tiden för att uppnå dessa resultat.

[ES] El software es un pilar fundamental en sectores como la banca, los videojuegos y la salud. Las pruebas de software son el método más empleado para garantizar la calidad y fiabilidad de estos sistemas. Al someter los sistemas a pruebas rigurosas y recopilar evidencias, este proceso permite tomar decisiones informadas sobre la calidad del producto. Los testers humanos son fundamentales en las pruebas de software, aportando intuición, creatividad y comprensión profunda. No obstante, los esfuerzos de los testers no deben centrarse en la ejecución manual y repetitiva de interacciones con el software, ya que ello implica una carga de trabajo excesiva, frustración y fatiga, lo que genera falta de atención y limitación en el tiempo de realización de pruebas. Scripted testing es un enfoque para automatizar pruebas de software con scripts secuenciales y oráculos de prueba, reduciendo el esfuerzo manual y ayudando a mantener la calidad del software al detectar fallos de regresión. Sin embargo, la necesidad de mantener un número incontrolable de scripts, requeridos para sistemas grandes y complejos, hace que las pruebas basadas en scripts resulten insuficientes. Scriptless testing es un enfoque automatizado complementario que no depende de scripts, sino que explora el software de manera dinámica mediante acciones no secuenciales. Este enfoque introduce aleatoriedad, lo que permite descubrir estados y fallos no identificados por pruebas manuales o con scripts. TESTAR es una herramienta de automatización de pruebas sin scripts de código abierto que ha demostrado ser eficaz en la cobertura y detección de fallos al probar aplicaciones de escritorio y web industriales, a través de la Interfaz Gráfica de Usuario (GUI). La arquitectura modular de TESTAR permite seguir mejorando la herramienta para: (1) conectarse con otros sistemas software complejos, (2) incorporar algoritmos exploratorios para mejorar la efectividad y eficiencia de las pruebas, (3) integrar diverso de oráculos de prueba para cubrir una amplia variedad de características y detectar sus fallos, y (4) ser validada en proyectos industriales y de código abierto bien conocidos. En esta tesis, hemos colaborado con socios industriales y proyectos de código abierto para mejorar TESTAR, integrando marcos tecnológicos para conectar con aplicaciones de escritorio, web, móvil y de Realidad Extendida (XR); añadiendo algoritmos exploratorios para mejorar la efectividad y eficiencia de los mecanismos de selección de acciones; e integrando nuevos oráculos de prueba para la detección automatizada de fallos. Estas nuevas extensiones se han evaluado con proyectos industriales y de código abierto bien establecidos. Nuestros resultados empíricos demuestran una mejora significativa en las capacidades de TESTAR. Primero, TESTAR puede conectarse y probar aplicaciones de escritorio, web, móvil y juegos de XR. Segundo, la implementación de procesos de inferencia de modelos de estado, un enfoque de inferencia distribuida, reglas de selección de acciones basadas en gramáticas y un algoritmo interactivo para juegos, mejoran significativamente la efectividad y la eficiencia de la exploración sin scripts. Tercero, la integración de oráculos de prueba y detección de cambios delta permiten a TESTAR detectar diversos tipos de fallos en el software e identificar cambios en la GUI entre versiones delta de la misma aplicación. Cuarto, TESTAR ha demostrado ser una solución complementaria a enfoques de pruebas manuales y scripts en la industria. Finalmente, hemos realizado una colaboración implementando un enfoque de Desarrollo Guiado por Comportamiento (BDD) para automatizar pruebas de regresión de un juego XR industrial. Esta tesis concluye que usar una herramienta scriptless como TESTAR puede mejorar significativamente el proceso de aseguramiento de la calidad (QA) en aplicaciones de escritorio, web, móviles y XR, al aumentar la cobertura y detectar fallos inesperados. / [CA] El programari és un pilar fonamental en sectors com la banca, els videojocs i la salut. Les proves de programari són el procés més utilitzat per assegurar la qualitat i la fiabilitat d'estos sistemes. Mitjançant proves rigoroses sobre els sistemes i la recopilació d'evidències, este procés permet prendre decisions informades sobre la qualitat d'un producte de programari. Els testers humans són fonamental en les proves de programari perquè aporten intuïció, creativitat i comprensió profunda. No obstant això, els seus esforços no haurien de destinar-se a l'execució d'interaccions manuals i repetitives amb el programari, ja que això genera càrregues de treball elevades, frustració i fatiga, cosa que comporta una falta d'atenció i de temps per realitzar les proves. Scripted testing és un enfocament per automatitzar l'execució de proves creant scripts seqüencials i oracles de prova, reduint l'esforç manual i ajudant a mantindre la qualitat del programari detectant fallades de regressió. No obstant això, la necessitat de mantindre un nombre incontrolable de scripts, per a sistemes grans i complexos, fan que la prova amb scripts siga un enfocament insuficient. Scriptless testing és un enfocament automatitzat complementari que no es basa en scripts, sinó que explora dinàmicament el programari amb accions no seqüencials. Este enfocament introduïx aleatorietat, la qual ajuda a descobrir estats i falles no identificats per les proves manuals o amb scripts. TESTAR és una eina d'automatització de proves sense scripts de codi obert que ha demostrat efectivitat en la cobertura i detecció de falles provant aplicacions industrials d'escriptori i web a través de la Interfície Gràfica d'Usuari (GUI). L'arquitectura modular de TESTAR permet a l'eina de millorar per a: (1) connectar-se amb altres sistemes de programari complexos, (2) incorporar algoritmes exploratoris per millorar l'efectivitat i l'eficiència de les proves, (3) integrar diversos oracles de prova per cobrir una àmplia varietat de característiques o descobrir els seus errors, i (4) ser validada en projectes industrials i de codi obert coneguts. En esta tesi, hem col·laborat amb socis industrials i projectes de codi obert per millorar TESTAR, integrant marcs tecnològics per connectar amb aplicacions d'escriptori, web, mòbils i de Realitat Estesa (XR); afegint algoritmes exploratoris per millorar l'efectivitat i l'eficiència dels mecanismes de selecció d'accions; i integrant nous oracles de prova per a la detecció automatitzada de fallades. Estes noves extensions s'han avaluat amb projectes industrials i de codi obert ben establits. Els nostres resultats empírics demostren una millora significativa en les capacitats de TESTAR. Primer, TESTAR pot connectar-se i provar aplicacions d'escriptori, web, mòbils i jocs de XR. En segon lloc, la implementació de processos d'inferència de models d'estat, un enfocament d'inferència distribuïda, regles de selecció d'accions basades en gramàtiques i un algoritme interactiu per a jocs milloren significativament l'efectivitat i l'eficiència de l'exploració sense scripts. En tercer lloc, la integració d'oracles de prova i la detecció de canvis delta permeten a TESTAR detectar diversos tipus de fallades en el programari i identificar canvis en la GUI entre versions delta de la mateixa aplicació. En quart lloc, TESTAR ha demostrat ser una solució complementària als enfocaments de proves manuals i amb scripts en la indústria. Finalment, hem col·laborat implementant un enfocament de Desenvolupament Guiat per Comportament (BDD) per automatitzar proves de regressió d'un joc XR industrial. Esta tesi conclou que utilitzar una eina scriptless com TESTAR pot millorar significativament el procés d'assegurament de la qualitat (QA) de les aplicacions d'escriptori, web, mòbils i XR augmentant la cobertura i detectant fallades inesperades. / [EN] Software is a fundamental pillar in sectors such as banking, video games, and healthcare. Software testing is the most commonly used method to ensure the quality and reliability of these systems. By rigorously exercising software systems and collecting evidence, testing enables informed decisions about a software product's quality. Human testers play a vital role in software testing, bringing intuition, creativity, and a deep understanding of real-world contexts. However, testers' efforts should not be misdirected to execute manual and repetitive software interactions, as this leads to excessive workload, frustration, and fatigue, resulting in decreased attention and limited testing time. Scripted testing automates testing through sequential scripts and test oracles, reducing manual effort and helping maintain software quality by detecting unintended regression failures. However, the need to maintain these scripts and the unmanageable number of scripts required for large and complex systems make scripted testing an insufficient approach. Scriptless testing is a complementary automated approach that does not rely on scripts but dynamically explores software with non-sequential actions. This approach introduces randomness, which helps to uncover states and failures not identified by manual or scripted testing. TESTAR is an open-source scriptless testing tool that has demonstrated its coverage and fault detection effectiveness when testing industrial desktop and web applications through the Graphical User Interface (GUI). The modular architecture of TESTAR enables the tool for further improvement to: (1) connect with other complex software systems, (2) incorporate additional exploratory algorithms for test effectiveness and efficiency, (3) integrate a diverse set of test oracles to cover a wide variety of features or discover its faults, and (4) be validated with industrial and well-known open-source projects. In this thesis, we conducted several research collaborations with industrial partners and open-source projects to improve TESTAR. We extended its interaction capabilities by integrating frameworks to test desktop, web, mobile, and eXtended Reality (XR) applications, added exploratory algorithms to improve the test effectiveness and efficiency of action selection mechanisms, and integrated a new set of test oracles for automated software assurance and fault detection. These novel extensions were evaluated with industrial applications and well-established open-source projects. Our empirical results demonstrate a significant improvement in TESTAR capabilities. First, TESTAR can successfully connect to and test various desktop, web, mobile, and XR game applications. Second, a state model inference processes, a distributed inference approach, grammar-based action selection rules, and an interactive algorithm for navigating game environments help significantly improve the effectiveness and efficiency of scriptless exploration. Third, the integration of online test oracles and offline delta change detection oracles enabled TESTAR to detect diverse types of software failures and identify and highlight GUI changes between delta versions of the same software application. Fourth, TESTAR has proved to be complementary to manual and scripted testing approaches in industrial organizations. Finally, we additionally collaborated to implement a Behavior-Driven Development (BDD) approach for automating regression tests of an XR industrial game system. This thesis concludes that using a scriptless testing tool like TESTAR can significantly improve the Quality Assurance (QA) testing process of desktop, web, mobile, and XR applications by increasing coverage and detecting unexpected failures. / This thesis has been developed at the Universitat Politècnica de València and has been funded by various European research projects: ITEA3 TESTOMAT project (TESTOMAT 16032), H2020 DEveloper COmpanion for Documented and annotatEd code Reference (DECODER 824231), H2020 Intelligent Verification/Validation for Extended Reality Based Systems (IV4XR 856716), ITEA3 Industrial-grade Verification and Validation of Evolving Systems (IVVES 18022), Erasmus+ European Innovation Alliance for Testing Education (ENACTEST 101055874), NWO OTP project Automated Unobtrusive Techniques for LINK- ing requirements and testing in agile software development (AUTOLINK 19521). / Pastor Ricos, F. (2025). Improving the QA Testing Process of Desktop, Web, Mobile, and XR Applications through Scriptless Testing using the TESTAR Tool [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/214481