Traditional test oracles have two problems. Firstly, several test oracles are needed for a single software program to perform different functions and maintaining a large number of test oracles is tedious and might be prone to errors. Secondly, testers usually test only the important criteria of a web application, since its time consuming to check with all the possible criteria. Ontologies have been used in a wide variety of domains and they have also been used in software testing. However, they have not been used for test oracle automation. The main idea of this thesis is to define a procedure for how ontology-based test oracle automation can be achieved for testing web applications and minimize the problems of traditional test oracles. The proposed procedure consists of the following steps: first, the expected results are stored in ontology A by running previous working version of the web application; second, the actual results are stored in ontology B by running the web application under test at runtime; and finally, the results of both ontology A and B are compared. This results in an automated test oracle comparator. Evaluation includes how the proposed procedure minimizes the traditional test oracle problems and by identifying the benefits of the defined procedure.

Financial markets use complex computer trading systems whose failures can cause serious economic damage, making reliability a major concern. Automated random testing has been shown to be useful in finding defects in these systems, but its inherent test oracle problem (automatic generation of the expected system output) is a drawback that has typically prevented its application on a larger scale. Two main tasks have been carried out in this thesis as a solution to the test oracle problem. First, an independent model of a real trading system based on constraint programming, a method for solving combinatorial problems, has been created. Then, the model has been integrated as a true test oracle in automated random tests. The test oracle maintains the expected state of an order book throughout a sequence of random trade order actions, and provides the expected output of every auction triggered in the order book by generating a corresponding constraint program that is solved with the aid of a constraint programming system. Constraint programming has allowed the development of an inexpensive, yet reliable test oracle. In 500 random test cases, the test oracle has detected two system failures. These failures correspond to defects that had been present for several years without being discovered neither by less complete oracles nor by the application of more systematic testing approaches. The main contributions of this thesis are: (1) empirical evidence of both the suitability of applying constraint programming to solve the test oracle problem and the effectiveness of true test oracles in random testing, and (2) a first attempt, as far as the author is aware, to model a non-theoretical continuous double auction using constraint programming. / Winner of the Swedish AI Society's prize for the best AI Master's Thesis 2010.

Embedded systems are present in many fields of application where failure may be critical. Such systems often possess characteristics that hampers the testing activity, as large amount of produced data and temporal requirements which must be specified and evaluated. There are tools that support the development of models for analysis and simulation still in the design stage. After being evaluated, a model may be used as basis to the implementation. In this case, it is important to ensure that the model is consistent with the specification. Otherwise, a divergence will be propagated to the final code. Therefore, the model must be tested prior to the codification. Simulink is a standard development and simulation tool for models of embedded systems. Its wide application in the industry has promoted the creation of free-software alternatives, as XCos. In the literature, there are researches which seek to improve the testing activity for Simulink-like models. The proposed solutions usually focus on test case selection strategies. However, little efforts have been directed to the oracle problem, that is, the difficulty in evaluating if an execution agrees with the specification. The objective of this doctorate proposal is to provide an oracle generation approach for Simulink-like models which addresses the characteristics previously summarized. Specifically, it is proposed a process, methods, procedures and a tool that enable the partially-automated generation of oracles for such models. As a main contribution, it is expected an improvement in the evaluation process of embedded systems in terms of quality, cost and time / Sistemas embarcados estão presentes em diversas áreas de aplicação em que falhas podem ser críticas. Tais sistemas frequentemente possuem características que tornam a fase de teste particularmente desafiadora, como a produção de grande quantidade de dados e requisitos temporais que precisam ser validados de acordo com a especificação. Existem ferramentas que auxiliam no desenvolvimento de modelos para análise e simulação do comportamento de sistemas embarcados ainda na fase de design. Após ser avaliado, o modelo pode ser usado como base para a implementação. Neste caso, deve-se buscar garantir que um modelo esteja de acordo com a especificação. Do contrário, tal divergência será propagada para a implementação. Portanto, e importante que o modelo seja testado antes da fase de implementação. Simulink e uma ferramenta-padrão de desenvolvimento e simulação de modelos de sistemas embarcados. Sua ampla aplicação na indústria incentivou a criação de alternativas de software livres como XCos. Na literatura, existem pesquisas que visam a aprimorar a atividade de teste de modelos Simulink-like. As soluções propostas geralmente focam em estratégias de seleção de casos de teste. Mas pouco esforço tem sido direcionado ao problema do oráculo, isto e, na dificuldade em avaliar se a execução está de acordo com a especificação. O objetivo desta proposta de doutorado é prover uma abordagem de geração de oráculos de teste para modelos simulink-like que contemple as características previamente resumidas. Especificamente, é proposto um processo, métodos, procedimentos e uma ferramenta que viabilizem a geração parcialmente automatizada de oráculos de teste para modelos Simulink-like. Como contribuição principal, é esperada a melhora da qualidade, custo e tempo do processo de validação de sistemas embarcados suportados por modelagem em Simulink e ferramentas similares

Kasdieninis žiniatinklio informacinių sistemų (toliau – IS) naudojimas paprastų vartotojų, verslo ir vyriausybinių organizacijų tarpe, reikalauja vis aukštesnės tokių IS kokybės. Žiniatinklio IS dažniausiai yra tokia programinė įranga, kurios sudėtingumas, dėl keleto apjungiamų technologijų, yra aukštas, o pokyčių greitis yra didelis. Vykdant palaikymo ir priežiūros darbus, gali atsirasti begalė klaidų, kurias iššaukia atlikti pakeitimai. Regresinio testavimo paskirtis yra užtikrinti, jog įdiegus programinės įrangos pakeitimus, tokius kaip naujo funkcionalumo pridėjimas ar jau egzistuojančio taisymas, ankstesnėje IS versijoje veikęs funkcionalumas veiks taip pat gerai ir naujojoje versijoje. Šiame darbe tiriami iš HTML žymų sudarytą IS atsaką galintys palyginti regresinio testavimo algoritmai (toliau – testų orakulai). HTML ãtsako kaip teksto lyginimas pateikia per daug klaidingai teigiamų (angl. false positives) rezultatų – testuotojas užverčiamas pranešimais apie netikras klaidas. Mūsų realizuotas testų orakulas HTML žymų atsaką nagrinėja kaip medžio struktūrą. Toks algoritmo veikimas leidžia tikrinti arba netikrinti aprašytas HTML semantines savybes, ko pasekoje yra sumažinamas klaidingai teigiamų ir klaidingai neigiamų rezultatų kiekis. Realizuotas testų orakulas nuo literatūroje aprašomų ir rinkoje siūlomų testų orakulų skiriasi tuo, jog nepraneša apie klaidą, kai HTML atsakas būna papildytas naujomis žymomis, kurios nekeičia pradinės medžio struktūros. Žiniatinklio IS... [toliau žr. visą tekstą] / Common usage of web information systems among consumers, business and governmental organizations require higher quality of such systems. Web information systems is such a software, which has high complexity because of the multiple technology usage and tends to rapid changes. Errors may occur because of the changes that have been made through maintenance period. Regression testing purpose is to ensure, that software changes such as new functionality or previously existed functionality modification, did not affect functionality, which was well working in the older version. In these master thesis we analyze test oracle comparators which can compare HTML response. HTML as a text comparison produces too many false positives. Meanwhile our test oracle comparator analyses HTML response as is was a tree structure. Thus implementation of the test oracle comparator algorithm enables opportunity to validate or not to validate described HTML semantic features. Therefore, we get fewer false positives and false negatives. Our implemented test oracle comparator differs from the ones described in literature or the ones proposed in market by capability not to warn tester about changes such as newly added HTML tag, which do not change overall tree structure. In this way the introduction of natural functionality growth is allowed. After improvement our test oracle comparator detects 90 % errors with the 90 % precision, whereas in the market existing test oracle comparator of the regression... [to full text]

Embedded systems are present in many fields of application where failure may be critical. Such systems often possess characteristics that hampers the testing activity, as large amount of produced data and temporal requirements which must be specified and evaluated. There are tools that support the development of models for analysis and simulation still in the design stage. After being evaluated, a model may be used as basis to the implementation. In this case, it is important to ensure that the model is consistent with the specification. Otherwise, a divergence will be propagated to the final code. Therefore, the model must be tested prior to the codification. Simulink is a standard development and simulation tool for models of embedded systems. Its wide application in the industry has promoted the creation of free-software alternatives, as XCos. In the literature, there are researches which seek to improve the testing activity for Simulink-like models. The proposed solutions usually focus on test case selection strategies. However, little efforts have been directed to the oracle problem, that is, the difficulty in evaluating if an execution agrees with the specification. The objective of this doctorate proposal is to provide an oracle generation approach for Simulink-like models which addresses the characteristics previously summarized. Specifically, it is proposed a process, methods, procedures and a tool that enable the partially-automated generation of oracles for such models. As a main contribution, it is expected an improvement in the evaluation process of embedded systems in terms of quality, cost and time / Sistemas embarcados estão presentes em diversas áreas de aplicação em que falhas podem ser críticas. Tais sistemas frequentemente possuem características que tornam a fase de teste particularmente desafiadora, como a produção de grande quantidade de dados e requisitos temporais que precisam ser validados de acordo com a especificação. Existem ferramentas que auxiliam no desenvolvimento de modelos para análise e simulação do comportamento de sistemas embarcados ainda na fase de design. Após ser avaliado, o modelo pode ser usado como base para a implementação. Neste caso, deve-se buscar garantir que um modelo esteja de acordo com a especificação. Do contrário, tal divergência será propagada para a implementação. Portanto, e importante que o modelo seja testado antes da fase de implementação. Simulink e uma ferramenta-padrão de desenvolvimento e simulação de modelos de sistemas embarcados. Sua ampla aplicação na indústria incentivou a criação de alternativas de software livres como XCos. Na literatura, existem pesquisas que visam a aprimorar a atividade de teste de modelos Simulink-like. As soluções propostas geralmente focam em estratégias de seleção de casos de teste. Mas pouco esforço tem sido direcionado ao problema do oráculo, isto e, na dificuldade em avaliar se a execução está de acordo com a especificação. O objetivo desta proposta de doutorado é prover uma abordagem de geração de oráculos de teste para modelos simulink-like que contemple as características previamente resumidas. Especificamente, é proposto um processo, métodos, procedimentos e uma ferramenta que viabilizem a geração parcialmente automatizada de oráculos de teste para modelos Simulink-like. Como contribuição principal, é esperada a melhora da qualidade, custo e tempo do processo de validação de sistemas embarcados suportados por modelagem em Simulink e ferramentas similares

The train represents a complex system, where every sub-system has an important role. If a subsystem doesn’t work how it should, the correctness of whole the train can be uncertain. To ensure that system works properly, we should test each sub-system individually and integrate them together in the whole system. Each of these subsystems consists of the different modules with different functionalities what should be tested. Testing of different functionalities often requires a different approach. For some functionalities, it is necessary domain knowledge from the human expert, such as classification of signals in different use cases in Propulsion and Controls (PPC) in Bombardier Transportation. Due to this reason, we need to simulate of using experts knowledge in the certain domain. We are investigating the use of machine learning techniques for solving this cases and creating system what will automatically classify different signals using the previous human knowledge. This case study is conducted in Bombardier Transportation (BT), Västerås in departments Train Control Management System (TCMS) and Propulsion and Controls (PPC), where data is collected, analyzed and evaluated. We proposed a method for solving the oracle problem based on machine learning approach for different for certain use case. Also, we explained different steps what can be used for solving the test oracle problem where signals are part of verdict process

Model-Based Testing has shown tangible benefits in previous research exploring widespread domains. Clearing systems in financial markets however are large software systems in which these approaches rarely have been explored. This paper in particular explores the Model-Based Testing approach using Modbat, a specialized tool based on extended finite-state machines, as a complementary testing approach on software clearing systems at Nasdaq. In particular, the approach is applied to functionalities of the system to test both its ability to find bugs undiscovered by traditional methods, as well as to measure its effectiveness and efficiency in creating reliable coverage in short amounts of time. In this study, we show that the model-based Modbat approach can achieve high code coverage with relatively little effort. 77 % of the production coverage was achieved with very few Modbat test runs. Additionally, only ∼ 9-14 % of the same lines of code, and a fraction of the time were required to achieve these results. From this, we conclude that Model-Based Testing approaches such as Modbat allow for significantly more efficient testing in terms of coverage over time compared to manual approaches. / Modellbaserad testning har visat på konkreta fördelar i tidigare forskning som utforskat omfattande områden. Clearingsystem inom finansmarknader är dock stora programvarusystem där dessa metoder sällan har utforskats. Denna arbete utforskar specifikt modellbaserad testning med användning av Modbat, ett specialiserat verktyg baserat på utökade ändliga automater, som ett kompletterande testningsmetodik för programvarurensningssystem på Nasdaq. Särskilt tillämpas metoden på specifika funktioner i systemet för att testa både dess förmåga att hitta buggar som inte upptäckts av traditionella metoder och att mäta dess effektivitet att skapa pålitlig täckning av kod på kort tid. I denna studie visar jag att den modellbaserade Modbat-metoden kan uppnå hög kodtäckning med relativt lite ansträngning. 77 % av produktionskoden täcktes med mycket få Modbat-testkörningar. Dessutom krävdes endast cirka 9-14 % av samma LOC, och en bråkdel av tiden för att uppnå dessa resultat.