Le test de conformité est le processus permettant de contrôler qu'un système possède un ensemble de propriétés souhaitées et se comporte conformément à certaines exigences prédéfinies. Dans ce contexte, les techniques de test passif sont utilisées lorsque le système sous test ne peut être interrompu ou l'accès aux interfaces du système est indisponible. Le test passif s'appuie sur l'observation de l'application pendant l'exécution, et la comparaison de l'observation avec le comportement attendu, défini à travers des propriétés de conformité. L'objectif de cette thèse est la définition d'une méthodologie de validation des protocoles communicants par test passif. Les approches existantes sont issues de travaux basés sur des spécifications à états finis ou de transitions étiquetées et comme tels, ils présument l'existence d'une relation de causalité entre les évènements observés dans la trace du système. Pour le traitement des protocoles basés sur des messages, comme le protocole SIP (fondamental pour les services IMS), telle causalité n’existe pas nécessairement et en outre, elle ne peut être déterminée que par la partie données du protocole. Étant donné que les techniques existantes sont optimisées pour traiter les parties de contrôle, ils présentent des limites pour les tests basés sur des parties de données: expressibilité réduite de propriétés de conformité, entre autres. Dans ce travail nous présentons une approche sur la base des messages et données pour traiter ces problèmes. Les observations dans une trace sont sous la forme de messages. Le comportement attendu est défini de manière ascendante, à partir des critères basés sur les relations entre les champs des données des messages. Des relations temporelles sont définies entre ces critères, par exemple, une propriété peut exiger que certains critères “doit être reconnu pour tous les messages dans la trace”. Notre approche permet d’exprimer des formules sur l’avenir et le passé de la trace, permettant de définir des critères plus généraux que ceux qui utilisent uniquement des parties de contrôle. Des problèmes liés à la satisfaction des propriétés et la déclaration des verdicts de conformité sont également discutés. Bien que l'observation d'un comportement défini comme une propriété est un indice de conformité, l'absence d'observation n'est pas nécessairement indicative d'une faute. Plusieurs solutions à ce problème ont été proposées et mises en œuvre dans ce travail. Enfin, notre travail présente des perspectives intéressantes en termes d'extensibilité pour la détection en ligne ou une expressivité améliorée, mais aussi car une approche basée sur des messages fournit une vision alternative aux techniques de test traditionnelles / Conformance testing is the process of checking that a system possesses a set of desired properties and behaves in accordance with some predefined requirements. In this context, passive testing techniques are used when the system under test cannot be interrupted or access to the system's interfaces is unavailable. Passive testing relies on the observation of the implementation during runtime, and the comparison of the observation with the expected behavior, defined through conformance properties. The objective of this thesis is to define a novel methodology to validate communicating protocols by passive testing. Existing approaches are derived from works with finite-state and labelled transition specifications and as such, they presume there exists a causality relation between the events observed in the implementation (the trace). When dealing with message-based protocols, such as the Session Initiation Protocol (fundamental for IMS services), such causality does not necessarily exist and furthermore, it may only be determined through data parts. Since existing techniques are optimized for dealing with control parts, they present limitations for testing based on data parts: reduced expressibility and succinctness of conformance properties, as well as problems to deal with satisfaction of properties including future conditions. In this work we present a message-based/data-centric approach for dealing with these issues. Observations in a trace are in the form of messages. Expected behavior is defined in a bottom-up fashion, starting from expected criteria that must be fulfilled by one or more messages, defined as constraints between the message data fields. Temporal relations by quantification over the criteria, e.g. a property may require that certain criteria "must be held for all messages in the trace". Our approach allows to express formulas about the future and past of the trace, allowing to define more general criteria than through control parts alone. Issues related to satisfaction of properties and declaration of conformance verdicts are also discussed here. Although observation of a behavior defined as a property is indication of conformance, lack of observation is not necessarily indicative of a fault. Several solutions to this issue have been proposed and implemented in this work. Finally, our work presents interesting perspectives, in terms of extensibility for online detection or improved expressiveness, but also since a message-based approach provides an alternative view to traditional testing techniques

Today, interoperability is the major challenge for e-Business and e-Government domains. The fundamental solution is the standardization in different levels of business-to-business interactions. However publishing standards alone are not enough to assure interoperability between products of different vendors. In this respect, testing and certification activities are very important to promote standard adoption, validate conformance and interoperability of the products and maintain correct information exchange. In e-Business collaborations, standards need to address different layers of interoperability stack / communication layer, business document layer and business process layer. Although there have been conformance and interoperability testing tools and initiatives for each one of these categories, there is currently no support for testing an integration of the above within a test scenario which is similar to real life use cases. Together with the integration of different layers of testing, testing process should be automated so that test case execution can be done at low cost, and repeated if required. In this theses, a highly adaptable and flexible Test Execution Model and a complementary XML based Test Description Language consisting of high level test constructs which can handle or simulate different parts or layers of the interoperability stack is designed. The computer interpretable test description language allow dynamic set up of test cases and provides flexibility to design, modify, maintain and extend the test functionality in contrast to a priori designed and hard coded test cases. The work presented in this thesis is a part of the TestBATN system supported by TUBITAK, TEYDEB Project No: 7070191.

Conformance testing is the process of checking that a system possesses a set of desired properties and behaves in accordance with some predefined requirements. In this context, passive testing techniques are used when the system under test cannot be interrupted or access to the system's interfaces is unavailable. Passive testing relies on the observation of the implementation during runtime, and the comparison of the observation with the expected behavior, defined through conformance properties. The objective of this thesis is to define a novel methodology to validate communicating protocols by passive testing. Existing approaches are derived from works with finite-state and labelled transition specifications and as such, they presume there exists a causality relation between the events observed in the implementation (the trace). When dealing with message-based protocols, such as the Session Initiation Protocol (fundamental for IMS services), such causality does not necessarily exist and furthermore, it may only be determined through data parts. Since existing techniques are optimized for dealing with control parts, they present limitations for testing based on data parts: reduced expressibility and succinctness of conformance properties, as well as problems to deal with satisfaction of properties including future conditions. In this work we present a message-based/data-centric approach for dealing with these issues. Observations in a trace are in the form of messages. Expected behavior is defined in a bottom-up fashion, starting from expected criteria that must be fulfilled by one or more messages, defined as constraints between the message data fields. Temporal relations by quantification over the criteria, e.g. a property may require that certain criteria "must be held for all messages in the trace". Our approach allows to express formulas about the future and past of the trace, allowing to define more general criteria than through control parts alone. Issues related to satisfaction of properties and declaration of conformance verdicts are also discussed here. Although observation of a behavior defined as a property is indication of conformance, lack of observation is not necessarily indicative of a fault. Several solutions to this issue have been proposed and implemented in this work. Finally, our work presents interesting perspectives, in terms of extensibility for online detection or improved expressiveness, but also since a message-based approach provides an alternative view to traditional testing techniques

Mobility occurs naturally in many distributed system applications such as telecommunications and electronic commerce. Mobility may reduce bandwidth consumption and coupling and increase flexibility. However, it seems that relatively little work has been done to support quality assurance techniques such as testing and verification of mobile systems. This thesis describes an approach for checking the conformance of a mobile, distributed application with respect to an executable model at runtime. The approach is based on kiltera -- a novel, high-level language supporting the description and execution of models of concurrent, mobile, distributed, and timed computation. The approach allows distributed, rather than centralized, monitoring. However, it makes very few assumptions about the platform that the mobile agent system is implemented in. We have implemented our approach and validated it using four case studies. Two of them are examples of mobile agent systems, the two others are implementations of distributed algorithms. Our approach was able to detect seeded faults in the implementations. To check the effectiveness and the efficiency of our approach more comprehensively a mutation-based evaluation framework has been implemented. In this framework a set of a new mutation operators for mobile agent systems has been identified in order to automatically generate and run a number of mutants programs and then evaluate the ability of our approach to detect these mutants. We found that our approach is very effective and efficient in killing the non-equivalent mutants. / Thesis (Ph.D, Computing) -- Queen's University, 2010-04-27 12:35:47.996

As técnicas de teste podem ser divididas, num primeiro nível, naquelas baseadas no código (caixa branca) e naquelas baseadas na especificação (caixa preta ou funcionais). Nenhuma delas é completa pois visam a identificar tipos diferentes de defeitos e a sua utilização em conjunto pode elevar o nível de confiabilidade das aplicações. Assim, tornam-se importantes estudos que contribuam para um melhor entendimento da relação existente entre técnicas funcionais e estruturais, como elas se complementam e como podem ser utilizadas em conjunto. Este trabalho está inserido no contexto do projeto PLAVIS (Plataforma para Validação e Integração de Software em Aplicações Espaciais), e tem como objetivo realizar um estudo comparativo entre as técnicas de geração de casos de teste funcionais (baseadas nas especificações formais) e os critérios estruturais baseados em fluxo de controle e fluxo de dados, aplicados nas implementações. Num contexto específico, esse estudo deve fornecer dados de como se relacionam essas duas técnicas (funcional e estrutural) gerando subsídios para sua utilização em conjunto. Num contexto mais amplo - o do projeto PLAVIS - visa a estabelecer uma estratégia de teste baseada em critérios funcionais e estruturais e que possam, juntamente com as ferramentas que dão suporte a eles, compor um ambiente de teste disponível à utilização em aplicações espaciais dentro do INPE / Testing techniques can be divided, in high level, in code-based ones (white box) and specification based ones (black box). None of them are complete as they intend to identify different kinds of faults. The use of them together can increase the application confidence level. Thus, it is important to investigate the relationship between structural testing techniques and functional testing techniques, how they complete themselves and how they can be used together. This paper was developed in the context of the Plavis (PLAtform of software Validation & Integration on Space systems) project. This project provides comparative studies between functional generation testing techniques (based on formal specifications) and structural generation testing techniques, such as control-flow and data-flow criteria, applied in the implementation. In a specific context, this study provides data about the relationship between these techniques and how they can be used together. In the context of the Plavis project, the goal is to provide a testing strategy, based on functional and structural criteria, and a set of tools, composing a testing environment to be used in Space Applications projects, at INPE

Conformance to standards and interoperability is a major challenge of today`s applications in all domains. Several standards have been developed and some are still under development to address the various layers in the interoperability stack. Conformance and interoperability testing involves checking whether the applications conform to the standards so that they can interoperate with other conformant systems. Only through testing, correct information exchange among applications can be guaranteed. National Health Information System (NHIS) of Turkey aims to provide a nation-wide infrastructure for sharing Electronic Health Records (EHRs). In order to guarantee the interoperability, the Ministry of Health (MoH), Turkey, developed an Implementation/Integration/Interoperability Profile based on HL7 standards. TestBATN - Testing Business Process, Application, Transport and Network Layers - is a domain and standards independent set of tools which can be used to test all of the layers of the interoperability stack, namely, the Communication Layer, Document Content Layer and the Business Process Layer. In this thesis work, the requirements for conformance and interoperability testing of the NHIS are analyzed, a testing approach is designated, test cases for several NHIS services are developed and deployed and a test execution control and monitoring environment within TestBATN is designed and implemented through the identified testing requirements. The work presented in this thesis is part of the TestBATN system supported by the T&Uuml / BiTAK TEYDEB Project No: 7070191 in addition by the Ministry of Health, Turkey.

Testing is the dominant verification technique used in the software industry today. The use of automatic test case execution increases, but the creation of test cases remains manual and thus error prone and expensive. To automate generation and selection of test cases, model-based testing techniques have been suggested. In this thesis two central problems in model-based testing are addressed: the problem of how to formally specify coverage criteria, and the problem of how to generate a test suite from a formal timed system model, such that the test suite satisfies a given coverage criterion. We use model checking techniques to explore the state-space of a model until a set of traces is found that together satisfy the coverage criterion. A key observation is that a coverage criterion can be viewed as consisting of a set of items, which we call coverage items. Each coverage item can be treated as a separate reachability problem. Based on our view of coverage items we define a language, in the form of parameterized observer automata, to formally describe coverage criteria. We show that the language is expressive enough to describe a variety of common coverage criteria described in the literature. Two algorithms for test case generation with observer automata are presented. The first algorithm returns a trace that satisfies all coverage items with a minimum cost. We use this algorithm to generate a test suite with minimal execution time. The second algorithm explores only states that may increase the already found set of coverage items. This algorithm works well together with observer automata. The developed techniques have been implemented in the tool CoVer. The tool has been used in a case study together with Ericsson where a WAP gateway has been tested. The case study shows that the techniques have industrial strength.

As técnicas de teste podem ser divididas, num primeiro nível, naquelas baseadas no código (caixa branca) e naquelas baseadas na especificação (caixa preta ou funcionais). Nenhuma delas é completa pois visam a identificar tipos diferentes de defeitos e a sua utilização em conjunto pode elevar o nível de confiabilidade das aplicações. Assim, tornam-se importantes estudos que contribuam para um melhor entendimento da relação existente entre técnicas funcionais e estruturais, como elas se complementam e como podem ser utilizadas em conjunto. Este trabalho está inserido no contexto do projeto PLAVIS (Plataforma para Validação e Integração de Software em Aplicações Espaciais), e tem como objetivo realizar um estudo comparativo entre as técnicas de geração de casos de teste funcionais (baseadas nas especificações formais) e os critérios estruturais baseados em fluxo de controle e fluxo de dados, aplicados nas implementações. Num contexto específico, esse estudo deve fornecer dados de como se relacionam essas duas técnicas (funcional e estrutural) gerando subsídios para sua utilização em conjunto. Num contexto mais amplo - o do projeto PLAVIS - visa a estabelecer uma estratégia de teste baseada em critérios funcionais e estruturais e que possam, juntamente com as ferramentas que dão suporte a eles, compor um ambiente de teste disponível à utilização em aplicações espaciais dentro do INPE / Testing techniques can be divided, in high level, in code-based ones (white box) and specification based ones (black box). None of them are complete as they intend to identify different kinds of faults. The use of them together can increase the application confidence level. Thus, it is important to investigate the relationship between structural testing techniques and functional testing techniques, how they complete themselves and how they can be used together. This paper was developed in the context of the Plavis (PLAtform of software Validation & Integration on Space systems) project. This project provides comparative studies between functional generation testing techniques (based on formal specifications) and structural generation testing techniques, such as control-flow and data-flow criteria, applied in the implementation. In a specific context, this study provides data about the relationship between these techniques and how they can be used together. In the context of the Plavis project, the goal is to provide a testing strategy, based on functional and structural criteria, and a set of tools, composing a testing environment to be used in Space Applications projects, at INPE

The adoption of Health Information Systems (HIS) by primary care clinics and practitioners has become a standard in the healthcare industry. This increase in HIS utilization enables the informatization and automation of many paper-based clinical workflows, such as clinical referrals, through systems interoperability. The healthcare industry defines several interoperability standards and mechanisms to support the exchange of data among HIS. For example, the health authorities, Interior Health and Northern Health, created the CDX system to provide interoperability for HIS across British Columbia using SOAP Web Services and HL7 Clinical Document Architecture (CDA) interoperability standards. The CDX interoperability allows HIS such as Electronic Medical Record (EMR) systems to exchange information with other HIS, such as patients clinical records, clinical notes and laboratory testing results. In addition, to ensure the EMR systems adhere to the CDX specification, these health authorities conduct conformance testing with the EMR vendors to certify the EMR systems. However, conformance testing can only cover a subset of the systems' specifications and a few use cases. Therefore, systems properties that are not closely associated with the systems (i.e. emergent properties) are hard, or even impractical, to assure using only conformance testing. System safety is one of these properties that are particularly significant for EMR systems because it deals with patient safety. A well-known approach for improving systems safety is through hazard analysis. For scenarios where the human factor is an essential part of the system, such as EMR systems, the System-Theoretic Process Analysis (STPA) is more appropriate than traditional hazard analysis techniques. In this work, we perform a hazard analysis using STPA on the CDX conformance profile in order to evaluate and improve the safety of the CDX system interoperability. In addition, we utilize and customize a tool named FASTEN to support and facilitate the analysis. To conclude, our analysis identified a number of new safety-related constraints and improved a few other already specified constraints. / Graduate

En raison de la complexité croissante des systèmes réactifs, le test est devenu un des éléments essentiels dans le processus de leur développement. Les tests de conformité avec des méthodes formelles concernent la correction du contrôle fonctionnel, par le biais des tests d'un système en boîte noire avec une spécification formelle du système. Les techniques passives de test sont utilisées lorsque l’exécution des systèmes testés ne peut pas être perturbée ou l'interface du système n'est pas fournie. Les techniques passives de test sont fondées sur l'observation et la vérification des propriétés du comportement d'un système sans interférer avec son fonctionnement normal. Les tests contribuent également à établir les comportements anormaux pendant l’exécution sur la base de l'observation de toute déviation d'un comportement prédéterminé. L'objectif principal de cette thèse est de présenter une nouvelle approche pour la mise en place des tests passifs fondés sur l'analyse des parties contrôle et données du système sous test. Au cours des dernières décennies, de nombreuses théories et outils ont été développés pour effectuer les tests de conformité. De fait, les spécifications ou les propriétés des systèmes réactifs sont souvent modélisés par différentes variantes de Labeled Transition Systems (LTS). Toutefois, ces méthodes ne prennent pas explicitement en compte les parties données du système, étant donné que le modèle sous-jacent de LTS n’est pas en mesure de le faire. Par conséquent, avec ces approches il est nécessaire d'énumérer les valeurs des données avant la modélisation du système. Cela conduit souvent au problème de l'explosion combinatoire de l'état-espace. Pour palier à cette limitation, nous avons étudié un modèle appelé Input-Output Symbolic Transition Systems (IOSTS) qui inclut explicitement toutes les données d'un système réactif. De nombreuses techniques de tests passives prennent uniquement en considération la partie du contrôle du système en négligeant les données, ou elles sont confrontées à une quantité énorme de données du processus. Dans notre approche, nous prenons en compte la partie contrôle et données en intégrant les concepts d'exécution symbolique et nous améliorons l'analyse de traces en introduisant des techniques de slicing des traces d’exécution. Les propriétés sont décrites à l'aide d'IOSTS et nous illustrons dans notre approche comment elles peuvent être testées sur l'exécution réelle des traces en optimisant l'analyse. Ces propriétés peuvent être conçues pour tester la conformité fonctionnelle d'un protocole ainsi que des propriétés de sécurité. Au-delà de l'approche théorique, nous avons développé un outil logiciel qui implémente les algorithmes présentés dans nos travaux. Enfin, comme preuve de concept de notre approche et de l'outil logiciel, nous avons appliqué les techniques à deux études de cas réels : le protocole SIP et le protocole Bluetooth / Due to the increasing complexity of reactive systems, testing has become an important part in the process of the development of such systems. Conformance testing with formal methods refers to checking functional correctness, by means of testing, of a black-box system under test with respect to a formal system specification, i.e., a specification given in a language with a formal semantics. In this aspect, passive testing techniques are used when the implementation under test cannot be disturbed or the system interface is not provided. Passive testing techniques are based on the observation and verification of properties on the behavior of a system without interfering with its normal operation, it also helps to observe abnormal behavior in the implementation under test on the basis of observing any deviation from the predefined behavior. The main objective of this thesis is to present a new approach to perform passive testing based on the analysis of the control and data part of the system under test. During the last decades, many theories and tools have been developed to perform conformance testing. However, in these theories, the specifications or properties of reactive systems are often modeled by different variants of Labeled Transition Systems (LTS). However, these methodologies do not explicitly take into account the system's data, since the underlying model of LTS are not able to do that. Hence, it is mandatory to enumerate the values of the data before modeling the system. This often results in the state-space explosion problem. To overcome this limitation, we have studied a model called Input-Output Symbolic Transition Systems (IOSTS) which explicitly includes all the data of a reactive system. Many passive testing techniques consider only the control part of the system and neglect data, or are confronted with an overwhelming amount of data values to process. In our approach, we consider control and data parts by integrating the concepts of symbolic execution and we improve trace analysis by introducing trace slicing techniques. Properties are described using Input Output Symbolic Transition Systems (IOSTSs) and we illustrate in our approach how they can be tested on real execution traces optimizing the trace analysis. These properties can be designed to test the functional conformance of a protocol as well as security properties. In addition to the theoretical approach, we have developed a software tool that implements the algorithms presented in this paper. Finally, as a proof of concept of our approach and tool we have applied the techniques to two real-life case studies: the SIP and Bluetooth protocol