An Instructable Software Agent (ISA) is a software agent that humans can teach through Natural Instruction Methods (NIMs)—methods humans naturally use to teach one another. Some examples of NIMs include giving demonstrations, guided practice sessions, and definitions of concepts. If software agents were instructable, humans would be able to impart knowledge to software systems though a more natural interface. In this dissertation, I address generating benchmarks for evaluating the learning ability of ISAs despite the important differences that may exist between human learners and ISAs. I first present three years of case studies uncovering the challenges of such a comparison and then make recommendations for future studies. The main contributions of this dissertation are 1. a theory of using humans to evaluate the learning ability of Instructable Software Agents (ISAs), 2. a refined method for developing curricula and benchmarks for evaluating ISAs, including a scalable lab configuration for performing human benchmarking and a suite of accompanying software tools, and 3. the case studies themselves, amounting to an in-depth ethnographic study of the issues involved in using humans to develop curricula and benchmarks for ISAs. / text

published_or_final_version / abstract / toc / Computer Science and Information Systems / Master / Master of Philosophy

published_or_final_version / Computer Science / Doctoral / Doctor of Philosophy

Τα τελευταία χρόνια, οι υπηρεσίες Δημόσιας Υγείας που προσφέρονται στους πολίτες χρειάζονται διαρκώς αυξανόμενη χρηματοδότηση. Αιτίες είναι η μείωση του λόγου εργαζομένων / συνταξιούχων, καθώς και η αύξηση του προσδόκιμου μέσου όρου ζωής. Συνεπώς, άμεση προτεραιότητα θα πρέπει να αποτελεί η διασφάλιση της αξιόπιστης διαχείρισης των πόρων που διατίθενται γι’ αυτόν το σκοπό. Στόχος αυτής της εργασίας είναι η υλοποίηση «έξυπνου» ηλεκτρονικού συστήματος, το οποίο: α) δίνει τη δυνατότητα ηλεκτρονικής καταχώρησης των συνταγών του συμβεβλημένου γιατρού προς στον ασφαλισμένο, β) δίνει στο χρήστη τη δυνατότητα ελέγχου για πιθανή απάτη / λανθασμένη συνταγογράφηση, αναφέροντας τις «ύποπτες» συνταγές, βάσει προκαθορισμένων κανόνων που θα έχουν καταχωρηθεί στο σύστημα. Παράλληλα, αναπτύχθηκε ένα ολοκληρωμένο σύστημα διαχείρισης των συνταγών, που προσφέρει τη δυνατότητα αναζητήσεων, εμφάνισης και εκτύπωσης των δεδομένων στη μορφή που θα ζητηθεί από το χρήστη. Η εφαρμογή αναπτύχθηκε στο σύστημα διαχείρισης βάσεων δεδομένων της Microsoft Access 2002, και για τον έλεγχο λειτουργίας της χρησιμοποιήθηκαν στοιχεία του ασφαλιστικού οργανισμού Τ.Σ.Α.Υ., μετά την αφαίρεση όλων των προσωπικών δεδομένων. / -

Software testing has been integrated within software development life cycle due to its importance in assuring software quality, software safety, and customers' satisfaction. However, problems in software testing are prominent among software developers as system grows in size and complexity. Dependency injection becomes an appealing solution for developers with its practicality to improve software design, improve testability, and enable mock testing technique. The study aims to discover the extent to which the dependency injection facilitates software design and software testing. In addition, the effect of mock practice on testing is also assessed. Metrics for investigation are defined and measured on various aspects of two systems. The two systems are selected and developed based on the same user requirements, development technologies, and methodologies. By comparing the two systems against the investigated metrics, we aim to reveal whether the dependency injection improve the code design. Then four test suites from both systems are evaluated in relation to testability. The results demonstrate that the dependency injection does not seem to improve the code design if comparing on the selected metrics. Even though it does not score better, its effect is evident in other software aspects. The testability of the two systems is similar and suffers from the same problem. Meanwhile, mock helps assist software testing and improve testability. The effect of mock technique can be witnessed, especially when it is applied with other test techniques. Explanations and discussions on these findings are addressed in the paper.

This master’s thesis, carried out at MAN Truck & Bus AG, presents a self-observational case study of the software  development  methodology  Test-driven  development  (TDD)  in  the  context  of  developing  a framework  for  human-machine  interface  concepts  in  commercial  vehicles.  Software  developers  are constantly looking for ways to improve productivity and the quality of their code. TDD has been said to do precisely this, but not many experience reports from new practitioners can be found, making it difficult to know what to expect when using it for the first time. This thesis focuses on the experiences of a beginner to the TDD practice and follows the development of the framework and the changes made to the design over time. The framework, consisting of a C++ server application and an Android client, was developed using TDD. Decisions, obstacles and general experiences from the development process are documented in this report with the aim of finding out how TDD works in practice for a beginner and how well the practice is suited for this particular kind of project. It was concluded that TDD seems to have both benefits and drawbacks, as it appears to facilitate lower coupling of the code and a more structured design, but also complicates the changing of public interfaces since the changes often also affect the test code. Subjectively perceived effects of the practice included that developer  focus  improved,  that  testing  actually  occurred  and  that  the  continuous  passing  of  tests  gave confidence and a sense of accomplishment to the developer. Furthermore it was concluded that experienced developers may reap more benefits from TDD whereas developers with little experience might have a harder time  adjusting  to  the  practice  and  may  not  see  any  notable  improvement  on  the  design.  The  observed developer in this study also found that TDD was difficult to get used to and that it would have been helpful to initially pair up with an experienced TDD practitioner to be properly guided through the first steps and to form good habits. Some  parts  of  the  framework  developed  were  left  out  of  the  TDD  because  of  complexity  and  time reasons, leading the suitability of the practice for similar frameworks to be judged as moderate. The areas that  induced  problematic  situations  were  multithreading,  networking  and  graphical  user  interfaces  which were all considered difficult to handle with TDD.

Thesis (Ph.D.)--Computing, Georgia Institute of Technology, 2009. / Committee Chair: Smaragdakis, Yannis; Committee Member: Dwyer, Matthew; Committee Member: Orso, Alessandro; Committee Member: Pande, Santosh; Committee Member: Rugaber, Spencer.

Currently, the increasing complexity of software and the short release cycles are becoming a challenge for testing software in an efficient and effective way. Traditionally, creating tests is done manually by engineers, which are then automatically or manually executed on the actual software. Manually creating test cases is a time-consuming effort. For the last couple of decades, researchers have proposed ways to improve this process by automating parts of the testing steps. One of these approaches that have gained a lot of attention is called Model-Based Testing (MBT). MBT has been suggested as a way of automatically creating tests at a lower cost. Nonetheless, it is not very well studied how MBT is actually applied in industrial contexts and how these tests compare to manually written ones. This is particularly true for industrial control software such as the one found in the train domain, where strict requirements on testing are in place. In this thesis, we investigate the literature and review the related work on case studies on the MBT use in industry and its evaluation. We perform a case study to evaluate MBT on a train control management system provided by Bombardier Transportation. We use Comformiq Creator MBT Tool to create models for functional requirements of a master controller function and generate test cases. We provide the result of the modeling approach as well as a comparison between automatic test cases created by Conformiq Creator and manual test cases written by industrial engineers at Bombardier Transportation using the following metrics: test coverage and time spent on testing. The results of this comparison suggest that test coverage of MBT is higher and test cases are more detailed than manual testing. Our results are not conclusive in regard to the cost of using MBT, mainly because this depends on different testing scenarios and how testing is performed. We show that MBT is a suitable approach for modeling the functional requirements of a realistic industrial control software function. In this thesis work, we focus on system-level testing. As future work, applying MBT on lower levels of testing can be a promising way forward for evaluation. In addition, the transformation of these test cases into executable test scripts and the possible problems needs to be investigated further.

O projeto, seja ele de software ou hardware, envolve uma série de atividades que, apesar das técnicas, ferramentas e métodos empregados, não estão livres de erros que podem levar ao mau funcionamento do produto final. Estes erros podem ocorrer durante a especificação do projeto, como também em estágios finais do desenvolvimento ou no processo de manufatura. A fim de minimizar prejuízos é necessário garantir a qualidade do sistema a partir da verificação do projeto, da validação de protótipo e do teste de fabricação. Por muito tempo o teste de hardware e o teste de software foram estudados como disciplinas completamente independentes. Porém, similaridades entre o desenvolvimento de software e o projeto de hardware já foram exploradas com sucesso em adaptações de técnicas originalmente desenvolvidas para um sendo utilizadas por outro. Um exemplo é a cobertura de código, que foi inicialmente desenvolvida para o teste de software, e agora é comumente utilizada na verificação de hardware. Visto que dispositivos são descritos em linguagem de descrição de hardware, e estas possuem características semelhantes às linguagens de programação, parece uma boa alternativa valer-se desta semelhança para utilizar os métodos propostos pela engenharia de software para garantir a qualidade do hardware desenvolvido. Utilizar tais métodos para gerar padrões de teste para dispositivos de hardware descritos em HDL (Hardware Description Language) e identificar nestas descrições características que, alteradas, aumentem a testabilidade dos mesmos, são os principais objetivos desta tese. / Both software and hardware designs require several tasks to increase reliability and ensure high quality of the final system. Although different techniques, tools and methods can be applied, error free products are difficult to be achieved. Errors may occur on design specification, on development stages and also during manufacturing process. To increase system quality and minimize costs it is mandatory to perform design verification, prototype validation and manufacturing test. For a long time hardware and software tests were studied as disciplines completely apart. However, similarities between software development and hardware design have already been explored successfully by adapting techniques originally developed for one of them, and applying to the other. For instance, code coverage concept and methods were firstly developed for software testing, but nowadays are commonly used in hardware verification. Due to the high similarity observed between software programming languages and hardware description languages (HDL), it seems to be a valuable approach applying software engineering techniques to help ensuring a high quality hardware device. Therefore, the main purpose of this thesis is to use such techniques to extract test patterns from HDL descriptions of hardware devices and to identify at these descriptions means to increase hardware testability.

O projeto, seja ele de software ou hardware, envolve uma série de atividades que, apesar das técnicas, ferramentas e métodos empregados, não estão livres de erros que podem levar ao mau funcionamento do produto final. Estes erros podem ocorrer durante a especificação do projeto, como também em estágios finais do desenvolvimento ou no processo de manufatura. A fim de minimizar prejuízos é necessário garantir a qualidade do sistema a partir da verificação do projeto, da validação de protótipo e do teste de fabricação. Por muito tempo o teste de hardware e o teste de software foram estudados como disciplinas completamente independentes. Porém, similaridades entre o desenvolvimento de software e o projeto de hardware já foram exploradas com sucesso em adaptações de técnicas originalmente desenvolvidas para um sendo utilizadas por outro. Um exemplo é a cobertura de código, que foi inicialmente desenvolvida para o teste de software, e agora é comumente utilizada na verificação de hardware. Visto que dispositivos são descritos em linguagem de descrição de hardware, e estas possuem características semelhantes às linguagens de programação, parece uma boa alternativa valer-se desta semelhança para utilizar os métodos propostos pela engenharia de software para garantir a qualidade do hardware desenvolvido. Utilizar tais métodos para gerar padrões de teste para dispositivos de hardware descritos em HDL (Hardware Description Language) e identificar nestas descrições características que, alteradas, aumentem a testabilidade dos mesmos, são os principais objetivos desta tese. / Both software and hardware designs require several tasks to increase reliability and ensure high quality of the final system. Although different techniques, tools and methods can be applied, error free products are difficult to be achieved. Errors may occur on design specification, on development stages and also during manufacturing process. To increase system quality and minimize costs it is mandatory to perform design verification, prototype validation and manufacturing test. For a long time hardware and software tests were studied as disciplines completely apart. However, similarities between software development and hardware design have already been explored successfully by adapting techniques originally developed for one of them, and applying to the other. For instance, code coverage concept and methods were firstly developed for software testing, but nowadays are commonly used in hardware verification. Due to the high similarity observed between software programming languages and hardware description languages (HDL), it seems to be a valuable approach applying software engineering techniques to help ensuring a high quality hardware device. Therefore, the main purpose of this thesis is to use such techniques to extract test patterns from HDL descriptions of hardware devices and to identify at these descriptions means to increase hardware testability.