The assessment of the performance of learners by means of benchmark experiments is established exercise. In practice, benchmark studies are a tool to compare the performance of several competing algorithms for a certain learning problem. Cross-validation or resampling techniques are commonly used to derive point estimates of the performances which are compared to identify algorithms with good properties. For several benchmarking problems, test procedures taking the variability of those point estimates into account have been suggested. Most of the recently proposed inference procedures are based on special variance estimators for the cross-validated performance. We introduce a theoretical framework for inference problems in benchmark experiments and show that standard statistical test procedures can be used to test for differences in the performances. The theory is based on well defined distributions of performance measures which can be compared with established tests. To demonstrate the usefulness in practice, the theoretical results are applied to benchmark studies in a supervised learning situation based on artificial and real-world data. / Series: Report Series SFB "Adaptive Information Systems and Modelling in Economics and Management Science"

Este trabalho propõe a extensão do critério Análise de Mutantes, originalmente desenvolvido para o teste de programas, para sua aplicação no teste de especificações do aspecto comportamental de Sistemas Reativos. Esses sistemas constituem hoje um componente fundamental em várias atividades humanas e, em geral, falhas nos mesmos podem envolver grandes riscos a vida ou ao patrimônio. Isso toma imprescindível um maior rigor no processo de desenvolvimento e, em particular, na atividade de teste, que é fundamentalmente baseada em simulação, não fornecendo critério que avalie essa atividade de forma quantitativa. A proposta aborda a aplicação da Análise de Mutantes na validação de especificações de Sistemas Reativos baseadas em três técnicas formais, que possuem apoio gráfico, mais utilizadas para este fim: Máquinas de Estados Finitos, Redes de Petri e Statecharts. Para a aplicação do critério nesse contexto, estabeleceu-se um paralelo entre os níveis de programa e de especificação, quanto as suas hipóteses básicas do programador competente e do efeito de acoplamento. Foram definidos os operadores de mutação para cada uma das três técnicas consideradas, além de critérios de mutação alternativa que visam a minimização no custo de aplicação do critério. Foram realizados, manualmente, dois experimentos com o objetivo de validar os mecanismos propostos. Um deles foi aplicado em Máquinas de Estados Finitos e o outro, em Redes de Petri. Os resultados mostram evidências do aspecto complementar do critério Análise de Mutantes em relação as formas disponíveis de teste de especificações existentes na literatura. Apresenta-se também um protótipo da ferramenta Proteum-RSIFSM que apóia a aplicação da Análise de Mutantes em Máquinas de Estados Finitos e discute-se a instanciação dessa ferramenta para apoiar a aplicação do critério nos contextos de Statecharts e Redes de Petri / Reactive Systems are a fundamental component in severa1 activities and failures in these systems may cause risks to life or financia1 losses. Thus, the development process and particularly the test activity must be carried on with extreme care. This work proposes the use of Mutation Analysis - a technique originally proposed at program level testing - in the context of testing and validation of specifications of Reactive Systems; three graphical techniques are considered: Finite State Machines, Petri Nets and Statecharts. Currently, the test of such specifications is mainly based on simulation. The relevance of a test adequacy assessment criterion for such activity has been recognized by many researchers and practitioners and constitutes the objective of this work. The application of Mutation Analysis in this context is based in the assumption that the basic hypothesis valid to the program level - the hypothesis of the competent programmer and coupling effect also hold to the specification level. Mutation operators are defined for the three specification techniques. Also, alternative criteria were defined aiming at reducing the cost of application of Mutation Analysis in this context. Two experiments were manually conducted in order to validate the proposed ideas. The results show evidences of the complementary aspects among existent methods and Mutation Analysis for testing specifications of Reactive Systems. A prototype of Proteum-RSIFSM, a to01 to support Mutation Analysis for Finite State Machines is presented and its extensions to Statecharts and to Petri Nets are discussed

Este trabalho propõe a extensão do critério Análise de Mutantes, originalmente desenvolvido para o teste de programas, para sua aplicação no teste de especificações do aspecto comportamental de Sistemas Reativos. Esses sistemas constituem hoje um componente fundamental em várias atividades humanas e, em geral, falhas nos mesmos podem envolver grandes riscos a vida ou ao patrimônio. Isso toma imprescindível um maior rigor no processo de desenvolvimento e, em particular, na atividade de teste, que é fundamentalmente baseada em simulação, não fornecendo critério que avalie essa atividade de forma quantitativa. A proposta aborda a aplicação da Análise de Mutantes na validação de especificações de Sistemas Reativos baseadas em três técnicas formais, que possuem apoio gráfico, mais utilizadas para este fim: Máquinas de Estados Finitos, Redes de Petri e Statecharts. Para a aplicação do critério nesse contexto, estabeleceu-se um paralelo entre os níveis de programa e de especificação, quanto as suas hipóteses básicas do programador competente e do efeito de acoplamento. Foram definidos os operadores de mutação para cada uma das três técnicas consideradas, além de critérios de mutação alternativa que visam a minimização no custo de aplicação do critério. Foram realizados, manualmente, dois experimentos com o objetivo de validar os mecanismos propostos. Um deles foi aplicado em Máquinas de Estados Finitos e o outro, em Redes de Petri. Os resultados mostram evidências do aspecto complementar do critério Análise de Mutantes em relação as formas disponíveis de teste de especificações existentes na literatura. Apresenta-se também um protótipo da ferramenta Proteum-RSIFSM que apóia a aplicação da Análise de Mutantes em Máquinas de Estados Finitos e discute-se a instanciação dessa ferramenta para apoiar a aplicação do critério nos contextos de Statecharts e Redes de Petri / Reactive Systems are a fundamental component in severa1 activities and failures in these systems may cause risks to life or financia1 losses. Thus, the development process and particularly the test activity must be carried on with extreme care. This work proposes the use of Mutation Analysis - a technique originally proposed at program level testing - in the context of testing and validation of specifications of Reactive Systems; three graphical techniques are considered: Finite State Machines, Petri Nets and Statecharts. Currently, the test of such specifications is mainly based on simulation. The relevance of a test adequacy assessment criterion for such activity has been recognized by many researchers and practitioners and constitutes the objective of this work. The application of Mutation Analysis in this context is based in the assumption that the basic hypothesis valid to the program level - the hypothesis of the competent programmer and coupling effect also hold to the specification level. Mutation operators are defined for the three specification techniques. Also, alternative criteria were defined aiming at reducing the cost of application of Mutation Analysis in this context. Two experiments were manually conducted in order to validate the proposed ideas. The results show evidences of the complementary aspects among existent methods and Mutation Analysis for testing specifications of Reactive Systems. A prototype of Proteum-RSIFSM, a to01 to support Mutation Analysis for Finite State Machines is presented and its extensions to Statecharts and to Petri Nets are discussed

O desenvolvimento de software baseado no paradigma Orientado a Objetos (OO) e baseado em componentes é uma realidade. Este trabalho trata de teste e validação dentro desse contexto. Observa-se que diversos trabalhos relacionados ao teste de programas OO vêm sendo desenvolvidos. Apesar de ser um ponto controverso, alguns pesquisadores consideram que critérios de teste desenvolvidos para o teste de programas procedimentais podem ser facilmente estendidos para o teste de programas OO, pelo menos para o teste de métodos. Ainda são poucas as iniciativas de estender critérios de fluxo de dados e critérios baseados em mutação, tradicionalmente utilizados no teste de programas procedimentais, para o teste de programas OO. O presente trabalho visa a contribuir na identificação e definição de recursos de teste e validação que possam ser utilizados no teste de programas OO, com ênfase nos critérios de teste baseados em fluxo de dados e em mutação, cobrindo as fases do teste de unidade e de integração. Além disso, para apoiar a aplicação desses critérios, é de fundamental importância o desenvolvimento de ferramentas automatizadas que permitam a realização de estudos comparativos e a transferência tecnológica para a indústria. Em suma, o presente trabalho traz contribuições teóricas, com a definição de critérios de teste; empírica, com a realização de estudos empíricos; e de automatização, com a definição e implementação de um ambiente integrado de teste e validação para programas OO. Exemplos são utilizados para ilustrar as idéias e ferramentas apresentadas neste trabalho. / The development of Object-Oriented (OO) and component-based software is a reality. This work investigates software testing and validation in this context. Several studies related with OO testing have been carried out. In spite of being a controversial point, some researchers state that the procedural testing criteria can be easily extended to OO program testing, for instance to the testing of methods. There are few initiatives aiming at applying data-flow and mutation-based criteria, traditionally used for procedural testing, to the test of OO programs. The present work aims at contributing to identify and define resources for OO program testing and validation, considering data-flow and mutation based testing criteria, involving the unit and integration testing phases. An integrated environment for testing and validation has been developed to support the application of these criteria. This environment provides means to comparative studies amongst the criteria and to technology transfer processes. This work provides contributions to the following testing perspectives: theoretical -- with the definition of testing criteria; empirical -- with the conduction of empirical studies; and to testing automation -- with the specification and implementation of an integrated environment for testing and validation of OO programs. Examples are provided to illustrate the ideas and tools presented in this work.

O desenvolvimento de software baseado no paradigma Orientado a Objetos (OO) e baseado em componentes é uma realidade. Este trabalho trata de teste e validação dentro desse contexto. Observa-se que diversos trabalhos relacionados ao teste de programas OO vêm sendo desenvolvidos. Apesar de ser um ponto controverso, alguns pesquisadores consideram que critérios de teste desenvolvidos para o teste de programas procedimentais podem ser facilmente estendidos para o teste de programas OO, pelo menos para o teste de métodos. Ainda são poucas as iniciativas de estender critérios de fluxo de dados e critérios baseados em mutação, tradicionalmente utilizados no teste de programas procedimentais, para o teste de programas OO. O presente trabalho visa a contribuir na identificação e definição de recursos de teste e validação que possam ser utilizados no teste de programas OO, com ênfase nos critérios de teste baseados em fluxo de dados e em mutação, cobrindo as fases do teste de unidade e de integração. Além disso, para apoiar a aplicação desses critérios, é de fundamental importância o desenvolvimento de ferramentas automatizadas que permitam a realização de estudos comparativos e a transferência tecnológica para a indústria. Em suma, o presente trabalho traz contribuições teóricas, com a definição de critérios de teste; empírica, com a realização de estudos empíricos; e de automatização, com a definição e implementação de um ambiente integrado de teste e validação para programas OO. Exemplos são utilizados para ilustrar as idéias e ferramentas apresentadas neste trabalho. / The development of Object-Oriented (OO) and component-based software is a reality. This work investigates software testing and validation in this context. Several studies related with OO testing have been carried out. In spite of being a controversial point, some researchers state that the procedural testing criteria can be easily extended to OO program testing, for instance to the testing of methods. There are few initiatives aiming at applying data-flow and mutation-based criteria, traditionally used for procedural testing, to the test of OO programs. The present work aims at contributing to identify and define resources for OO program testing and validation, considering data-flow and mutation based testing criteria, involving the unit and integration testing phases. An integrated environment for testing and validation has been developed to support the application of these criteria. This environment provides means to comparative studies amongst the criteria and to technology transfer processes. This work provides contributions to the following testing perspectives: theoretical -- with the definition of testing criteria; empirical -- with the conduction of empirical studies; and to testing automation -- with the specification and implementation of an integrated environment for testing and validation of OO programs. Examples are provided to illustrate the ideas and tools presented in this work.

As part of the first agreement between the French (CNES) and the Chinese (CNSA) space agencies in terms of space exploration, the DORN instrument will fly on board the Chinese Chang'E 6 mission in 2024. It aims to measure the amount of Radon escaping from the lunar soil to better understand what governs the concentration of this noble gas which contributes to the endogenous origin of the lunar exosphere. This instrument is an alpha-particle spectrometer able to detect the radioactive decay of two Radon isotopes and their progeny. These emissions act as a tracing signal for the processes happening in the lunar sub-surface. During the Assembly, Integration & Testing phase of the Flight Model of the instrument, different scientific concerns must be addressed to characterise the behaviour of the instrument before it is commissioned to China for its final tests and integration into the lander before the mission. This means that all the measuring process, since the alpha-particle deposits its energy into the detector until it is counted as an event in a spectral data product, has to be meticulously analysed and assessed to verify if it is compliant with the instrument's scientific requirements. Under this context, several data analysis tools have been developed to retrieve the behaviour characteristics of the instrument. Firstly, due to the changing conditions that a space mission to the lunar environment undergoes, it is needed to perform an in-flight calibration at every measurement. To this regard, a program able to obtain the calibration parameters at several temperatures within its operative range along with the spectral resolution of each detector has been developed and used during the Thermal Vacuum tests held at CNES. Secondly, the numerical pipeline, responsible for processing and storing onboard data, must undergo stress testing and be exposed to conditions resembling the lunar environment. As a result, the software's capabilities enabling the instrument to derive particle energy and eliminate undesired detections have undergone thorough testing. These tools have been used to verify the compliance of the instrument with its respective scientific requirements as well as to validate its current software version. They not only have played an important role in finalising the development of the instrument in France but will be of great use during the tests done in China and during the space operations on the Moon. / Dans le cadre du premier accord entre les agences spatiales française (CNES) et chinoise (CNSA) en matière d'exploration spatiale, l'instrument DORN volera à bord de la mission chinoise Chang'E 6 en 2024. Il vise à mesurer la quantité de Radon s'échappant du sol lunaire afin de mieux comprendre ce qui gouverne la concentration de ce gaz noble qui contribue à l'origine endogène de l'exosphère lunaire. Cet instrument est un spectromètre à particules alpha capable de détecter la désintégration radioactive de deux isotopes du radon et de ses descendants. Ces émissions agissent comme un signal de traçage des processus qui se déroulent dans la sub-surface lunaire. Au cours de la phase d'assemblage, d'intégration et d'essai du modèle de vol de l'instrument, différentes questions scientifiques doivent être abordées pour caractériser le comportement de l'instrument avant qu'il ne soit envoyé en Chine pour les essais finaux et l'intégration dans l'atterrisseur avant la mission. Cela signifie que l'ensemble du processus de mesure, depuis le dépôt de l'énergie de la particule alpha dans le détecteur jusqu'à son comptage en tant qu'événement dans un produit de données spectrales, doit être méticuleusement analysé et évalué pour vérifier s'il est conforme aux exigences scientifiques de l'instrument. Dans ce contexte, plusieurs outils d'analyse de données ont été développés pour retrouver les caractéristiques de comportement de l'instrument. Tout d'abord, en raison des conditions changeantes d'une mission spatiale dans l'environnement lunaire, il est nécessaire d'effectuer un étalonnage en vol à chaque mesure. À cet égard, un programme capable d'obtenir les paramètres d'étalonnage à plusieurs températures dans sa gamme opérationnelle ainsi que la résolution spectrale de chaque détecteur a été utilisé pendant les tests de vide thermique qui se sont déroulés au CNES. Deuxièmement, le pipeline numérique, chargé du traitement et du stockage des données à bord, doit subir des tests de contrainte et être exposé à des conditions ressemblant à l'environnement lunaire. En conséquence, les capacités logicielles permettant à l'instrument de déduire l'énergie des particules et d'éliminer les détections indésirables ont fait l'objet de tests approfondis. Ces outils ont été utilisés pour vérifier la conformité de l'instrument avec ses exigences scientifiques respectives et pour valider la version actuelle de son logiciel. Ils ont non seulement joué un rôle important dans la finalisation du développement de l'instrument en France, mais seront également très utiles lors des tests effectués en Chine et lors des opérations spatiales sur la Lune.

This research undertakes the problem of vibration control of vehicular and structural systems using intelligent materials and controllable devices. Advanced modeling tools validated with experimental test data are developed to help with understanding the fundamentals as well as advanced and novel applications of smart and conventional suspension systems. The project can be divided into two major parts. The first part is focused on development of novel smart suspensions using Magneto-Rheological (MR) fluids in unique configurations in order to improve efficiency, controllability, and safety of today's vehicles. In this part of the research, attention is paid to fundamentals as well as advanced applications of MR technology. Extensive rheological studies, both theoretical and experimental, are performed to understand the basic behaviors of MR fluids as complex non-Newtonian fluids in novel applications. Using the knowledge obtained from fundamental studies of MR fluids, unique application concepts are investigated that lead to design, development, and experimental testing of two new classes of smart devices: MR Hybrid Dampers and MR Squeeze Mounts. Multiple generations of these devices are built and tested as proof of concept prototypes. Advanced physics-based mathematical models are developed for these devices. Experimental test data are used to validate the models and great agreement is obtained. The models are used as design tools at preliminary as well as detailed design stages of device development. The significant finding in this part of the research is that MR fluids can deliver a much larger window of controllable force in squeeze mode compared to shear and valve modes which can be used in various applications. The second part of the research is devoted to the development of innovative design tools for suspension design and tuning. Various components of suspension systems are studied and modeled using a new physics-based modeling approach. The component of main interest is the shim stack assembly in hydraulic dampers which is modeled using energy and variational methods. A major finding is that the shims should be modeled individually in order to represent the sliding effects properly when the shim stack is deflected. Next, the individual component models are integrated into a full suspension model. This model is then used as a tool for suspension design, synthesis, and tuning. Using this design tool, suspension engineers in manufacturing companies and other industrial sections can easily perform parametric studies without the need to carry out time consuming and expensive field and laboratory tests. / Ph. D.