Automated grading systems provide feedback to computer science students in a variety of ways, but often focus on incorrect program behaviors. These tools will provide indications of test case failures or runtime errors, but without debugging skills, students often become frus- trated when they don't know where to start. They know their code has defects, but finding the problem may be beyond their experience, especially for beginners. An additional concern is balancing the need to provide enough direction to be useful, without giving the student so much direction that they are effectively given the answer. Drawing on the experiences of the software engineering community, in this work we apply a technique called statistical fault location (SFL) to student program assignments. Using the GZoltar software tool, we applied this technique to a set of previously-submitted student assignments gathered from students in our introductory CS course, CS 1114: Introduction to Software Design. After a manual inspection of the student code, this exercise demonstrated that the SFL technique identifies the defective method in the first three most suspicious methods in the student's code 90% of the time. We then developed a plug-in for Web-CAT to allow new student submissions to be evaluated with the GZoltar SFL system. Additionally, we developed a tool to create a heat map visualization to show the results of the SFL evaluation overlaid on the student's source code. We deployed this toolset for use in CS 1114 in Fall 2017. We then surveyed the students about their perceptions of the utility of the visualization for helping them understand how to find and correct the defects in their code, versus not having access to the heat map. Their responses led to refinements in our presentation of the feedback. We also evaluated the performance of CS 1114 classes from two semesters and discovered that having the heat maps led to more frequent incremental improvements in their code, as well as reaching their highest correctness score on instructor-provided tests more quickly than students that did not have access to the heat maps. Finally, we suggest several directions for future enhancements to the feedback interface. / Doctor of Philosophy / Automated grading systems provide feedback to computer science students in a variety of ways, but often focus on incorrect program behaviors. These tools will provide indications of test case failures or runtime errors, but without debugging skills, students often become frus- trated when they don't know where to start. They know their code has defects, but finding the problem may be beyond their experience, especially for beginners. An additional concern is balancing the need to provide enough direction to be useful, without giving the student so much direction that they are effectively given the answer. Drawing on the experiences of the software engineering community, in this work we apply a technique called statistical fault location (SFL) to student program assignments. Using the GZoltar software tool, we applied this technique to a set of previously-submitted student assignments gathered from students in our introductory CS course, CS 1114: Introduction to Software Design. After a manual inspection of the student code, this exercise demonstrated that the SFL technique identifies the defective method in the first three most suspicious methods in the student's code 90% of the time. We then developed a plug-in for Web-CAT to allow new student submissions to be evaluated with the GZoltar SFL system. Additionally, we developed a tool to create a heat map visualization to show the results of the SFL evaluation overlaid on the student's source code. We deployed this toolset for use in CS 1114 in Fall 2017. We then surveyed the students about their perceptions of the utility of the visualization for helping them understand how to find and correct the defects in their code, versus not having access to the heat map. Their responses led to refinements in our presentation of the feedback. We also evaluated the performance of CS 1114 classes from two semesters and discovered that having the heat maps led to more frequent incremental improvements in their code, as well as reaching their highest correctness score on instructor-provided tests more quickly than students that did not have access to the heat maps. Finally, we suggest several directions for future enhancements to the feedback interface.

Developers often spend much of their valuable development time on software debugging and bug finding. In addition, software defects cost software industry as a whole hundreds or even a trillion of US dollars. As a result, many fault localization (FL) techniques for localizing bugs automatically, have been proposed. Despite its popularity, adopting FL in industrial environments has been impractical due to its undesirable accuracy and high runtime overhead cost. Motivated by the real-world challenges of FL applicability, this dissertation addresses these issues by proposing two main enhancements to the existing FL. First, it explores different strategies to combine a variety of program execution information with Information Retrieval-based fault localization (IRFL) techniques to increase FL's accuracy. Second, this dissertation research invents and experiments with the unconventional techniques of Instant Fault Localization (IFL) using the innovative concept of triggering modes. Our empirical evaluations of the proposed approaches on various types of bugs in a real software development environment shows that both FL's accuracy is increased and runtime is reduced significantly. We find that execution information helps increase IRFL's Top-10 by 17–33% at the class level, and 62–100% at the method level. Another finding is that IFL achieves as much as 100% runtime cost reduction while gaining comparable or better accuracy. For example, on single-location bugs, IFL scores 73% MAP, compared with 56% of the conventional approach. For multi-location bugs, IFL's Top-1 performance on real bugs is 22%, just right below 24% that of the existing FL approaches. We hope the results and findings from this dissertation help make the adaptation of FL in the real-world industry more practical and prevalent. / Doctor of Philosophy / In software engineering, fault localization (FL) is a popular technique to automatically find software bugs, which cost a huge loss of hundreds of billions of US dollars on the software industry. Despite its high demanding and popularity, adopting FL in industrial software companies remains impractical. To help resolve this applicability problem, this dissertation proposed enhanced techniques to localize bugs more accurately and with less overhead runtime expenses. As a result, FL becomes more practical and efficient for software companies.

Debugging is one of the most time-consuming activities in software development. Several fault localization techniques have been proposed in the last years, aiming to reduce development costs. A promising approach, called Spectrum-based Fault localization (SFL), consists of techniques that provide a list of suspicious program elements (e.g., statements, basic blocks, methods) more likely to be faulty. Developers should inspect a suspiciousness list to search for faults. However, these fault localization techniques are not yet used in practice. These techniques are based on assumptions about the developer\'s behavior when inspecting such lists that may not hold in practice. A developer is supposed to inspect an SFL list from the most to the least suspicious program elements (e.g., statements) until reaching the faulty one. This assumption leads to some implications: the techniques are assessed only by the position of a bug in a list; a bug is deemed as found when the faulty element is reached. SFL techniques should pinpoint the faulty program elements among the first picks to be useful in practice. Most techniques use ranking metrics to assign suspiciousness values to program elements executed by the tests. These ranking metrics have presented similar modest results, which indicates the need for different strategies to improve the effectiveness of SFL. Moreover, most techniques use only control-flow spectra due to the high execution costs associated with other spectra, such as data-flow. Also, little research has investigated the use of SFL techniques by practitioners. Understanding how developers use SFL may help to clarify the theoretical assumptions about their behavior, which in turn can collaborate with the proposal of techniques more feasible for practical use. Therefore, user studies are a valuable tool for the development of the area. The goal of this thesis research was to propose strategies to improve spectrum-based fault localization, focusing on its practical use. This thesis presents the following contributions. First, we investigate strategies to provide contextual information for SFL. These strategies helped to reduce the amount of code to be inspected until reaching the faults. Second, we carried out a user study to understand how developers use SFL in practice. The results show that developers can benefit from SFL to locate bugs. Third, we explore the use of data-flow spectrum for SFL. Data-flow spectrum singles out faults significantly better than control-flow spectrum, improving the fault localization effectiveness. / Depuração é uma das atividades mais custosas durante o desenvolvimento de programas. Diversas técnicas de localização de defeitos têm sido propostas nos últimos anos com o objetivo de reduzir custos de desenvolvimento. Uma abordagem promissora, chamada Localização de Defeitos baseada em Espectro (LDE), é formada por técnicas que fornecem listas contendo elementos de código (comandos, blocos básicos, métodos) mais suspeitos de conter defeitos. Desenvolvedores deveriam inspecionar uma lista de suspeição para procurar por defeitos. No entanto, essas técnicas de localização de defeitos ainda não são usadas na prática. Essas técnicas baseiam-se em suposições sobre o comportamento de desenvolvedores durante a inspeção de tais listas que podem não ocorrer na prática. Um desenvolvedor supostamente inspeciona uma lista de LDE a partir do elemento mais suspeito para o menos suspeito até atingir o elemento defeituoso. Essa suposição leva a algumas implicações: as técnicas são avaliadas somente pela posição dos defeitos nas listas; um defeito é considerado como encontrado quando o elemento defeituoso é atingido. Técnicas de LDE deveriam posicionar os elementos de código defeituosos entre as primeiras posições para serem úteis na prática. A maioria das técnicas usa métricas de ranqueamento para atribuir valores de suspeição aos elementos executados pelos testes. Essas métricas de ranqueamento têm apresentado resultados semelhantes, o que indica a necessidade de estratégias diferentes para melhorar a eficácia de LDE. Além disso, a maioria das técnicas usa somente espectros de fluxo de controle devido ao alto custo de execução associado a outros espectros, tais como fluxo de dados. Também, poucas pesquisas têm investigado o uso de técnicas de LDE por programadores. Entender como desenvolvedores usam LDE pode ajudar a esclarecer as suposições teóricas sobre seu comportamento, o que por sua vez pode para colaborar para a proposição de técnicas mais viáveis para uso prático. Portanto, estudos com usuários são importantes para o desenvolvimento da área. O objetivo desta pesquisa de doutorado foi propor estratégias para melhorar a localização de defeitos baseada em espectro focando em seu uso prático. Esta tese apresenta as seguintes contribuições originais. Primeiro, nós investigamos estratégias para fornecer informação de contexto para LDE. Essas estratégias ajudaram a reduzir quantidade de código a ser inspecionado até atingir os defeitos. Segundo, nós realizamos um estudo com usuários para entender como desenvolvedores usam LDE na prática. Os resultados mostram que desenvolvedores podem beneficiar-se de LDE para localizar defeitos. Terceiro, nós exploramos o uso de espectros de fluxo de dados para LDE. Mostramos que o espectro de fluxo de dados seleciona defeitos significamente melhor que espectro de fluxo de controle, aumentando a eficácia de localização de defeitos.

Debugging is one of the most time-consuming activities in software development. Several fault localization techniques have been proposed in the last years, aiming to reduce development costs. A promising approach, called Spectrum-based Fault localization (SFL), consists of techniques that provide a list of suspicious program elements (e.g., statements, basic blocks, methods) more likely to be faulty. Developers should inspect a suspiciousness list to search for faults. However, these fault localization techniques are not yet used in practice. These techniques are based on assumptions about the developer\'s behavior when inspecting such lists that may not hold in practice. A developer is supposed to inspect an SFL list from the most to the least suspicious program elements (e.g., statements) until reaching the faulty one. This assumption leads to some implications: the techniques are assessed only by the position of a bug in a list; a bug is deemed as found when the faulty element is reached. SFL techniques should pinpoint the faulty program elements among the first picks to be useful in practice. Most techniques use ranking metrics to assign suspiciousness values to program elements executed by the tests. These ranking metrics have presented similar modest results, which indicates the need for different strategies to improve the effectiveness of SFL. Moreover, most techniques use only control-flow spectra due to the high execution costs associated with other spectra, such as data-flow. Also, little research has investigated the use of SFL techniques by practitioners. Understanding how developers use SFL may help to clarify the theoretical assumptions about their behavior, which in turn can collaborate with the proposal of techniques more feasible for practical use. Therefore, user studies are a valuable tool for the development of the area. The goal of this thesis research was to propose strategies to improve spectrum-based fault localization, focusing on its practical use. This thesis presents the following contributions. First, we investigate strategies to provide contextual information for SFL. These strategies helped to reduce the amount of code to be inspected until reaching the faults. Second, we carried out a user study to understand how developers use SFL in practice. The results show that developers can benefit from SFL to locate bugs. Third, we explore the use of data-flow spectrum for SFL. Data-flow spectrum singles out faults significantly better than control-flow spectrum, improving the fault localization effectiveness. / Depuração é uma das atividades mais custosas durante o desenvolvimento de programas. Diversas técnicas de localização de defeitos têm sido propostas nos últimos anos com o objetivo de reduzir custos de desenvolvimento. Uma abordagem promissora, chamada Localização de Defeitos baseada em Espectro (LDE), é formada por técnicas que fornecem listas contendo elementos de código (comandos, blocos básicos, métodos) mais suspeitos de conter defeitos. Desenvolvedores deveriam inspecionar uma lista de suspeição para procurar por defeitos. No entanto, essas técnicas de localização de defeitos ainda não são usadas na prática. Essas técnicas baseiam-se em suposições sobre o comportamento de desenvolvedores durante a inspeção de tais listas que podem não ocorrer na prática. Um desenvolvedor supostamente inspeciona uma lista de LDE a partir do elemento mais suspeito para o menos suspeito até atingir o elemento defeituoso. Essa suposição leva a algumas implicações: as técnicas são avaliadas somente pela posição dos defeitos nas listas; um defeito é considerado como encontrado quando o elemento defeituoso é atingido. Técnicas de LDE deveriam posicionar os elementos de código defeituosos entre as primeiras posições para serem úteis na prática. A maioria das técnicas usa métricas de ranqueamento para atribuir valores de suspeição aos elementos executados pelos testes. Essas métricas de ranqueamento têm apresentado resultados semelhantes, o que indica a necessidade de estratégias diferentes para melhorar a eficácia de LDE. Além disso, a maioria das técnicas usa somente espectros de fluxo de controle devido ao alto custo de execução associado a outros espectros, tais como fluxo de dados. Também, poucas pesquisas têm investigado o uso de técnicas de LDE por programadores. Entender como desenvolvedores usam LDE pode ajudar a esclarecer as suposições teóricas sobre seu comportamento, o que por sua vez pode para colaborar para a proposição de técnicas mais viáveis para uso prático. Portanto, estudos com usuários são importantes para o desenvolvimento da área. O objetivo desta pesquisa de doutorado foi propor estratégias para melhorar a localização de defeitos baseada em espectro focando em seu uso prático. Esta tese apresenta as seguintes contribuições originais. Primeiro, nós investigamos estratégias para fornecer informação de contexto para LDE. Essas estratégias ajudaram a reduzir quantidade de código a ser inspecionado até atingir os defeitos. Segundo, nós realizamos um estudo com usuários para entender como desenvolvedores usam LDE na prática. Os resultados mostram que desenvolvedores podem beneficiar-se de LDE para localizar defeitos. Terceiro, nós exploramos o uso de espectros de fluxo de dados para LDE. Mostramos que o espectro de fluxo de dados seleciona defeitos significamente melhor que espectro de fluxo de controle, aumentando a eficácia de localização de defeitos.

Submitted by Franciele Moreira (francielemoreyra@gmail.com) on 2018-10-30T13:30:59Z No. of bitstreams: 2 Dissertação - Diogo Machado de Freitas - 2018.pdf: 1477764 bytes, checksum: 73759c5ece96bf48ffd4d698f14026b9 (MD5) license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) / Approved for entry into archive by Luciana Ferreira (lucgeral@gmail.com) on 2018-10-30T13:41:38Z (GMT) No. of bitstreams: 2 Dissertação - Diogo Machado de Freitas - 2018.pdf: 1477764 bytes, checksum: 73759c5ece96bf48ffd4d698f14026b9 (MD5) license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) / Made available in DSpace on 2018-10-30T13:41:38Z (GMT). No. of bitstreams: 2 Dissertação - Diogo Machado de Freitas - 2018.pdf: 1477764 bytes, checksum: 73759c5ece96bf48ffd4d698f14026b9 (MD5) license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) Previous issue date: 2018-09-24 / Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES / Fault Localization is one stage of the software life cycle, which demands important resources such as time and effort spent on a project. There are several initiatives towards the automation of the fault localization process and the reduction of the associated resources. Many techniques are based on heuristics that use information obtained (spectrum) from the execution of test cases, in order to measure the suspiciousness of each program element to be defective. Spectrum data generally refers to code coverage and test results (positive or negative). The present work presents two approaches based on the Genetic Programming algorithm for the problem of Fault Localization: a method to compose a new heuristic from a set of existing ones; and a method for constructing heuristics based on data from program mutation analysis. The innovative aspects of both methods refer to the joint investigation of: (i) specialization of heuristics for certain programs; (ii) application of an evolutionary approach to the generation of heuristics with non-linear equations; (iii) creation of heuristics based on the combination of traditional heuristics; (iv) use of coverage and mutation spectra extracted from the test activity; (v) analyzing and comparing the efficacy of methods that use coverage and mutation spectra for fault localization; and (vi) quality analysis of the mutation spectra as a data source for fault localization. The results have pointed to the competitiveness of both approaches in their contexts. / Localização de Defeitos é uma etapa do ciclo de vida de software, que demanda recursos importantes tais como o tempo e o esforço gastos em um projeto. Existem diversas iniciativas na direção da automação do processo de localização de defeitos e da redução dos recursos associados. Muitas técnicas são baseadas heurísticas que utilizam informação obtida (espectro) a partir da execução de casos de teste, visando a medir a suspeita de cada elemento de programa para ser defeituoso. Os dados de espectro referem-se, em geral, à cobertura de código e aos resultados dos teste (positivo ou negativo). O presente trabalho apresenta duas abordagens baseadas no algoritmo Programação Genética para o problema de Localização de Defeitos: um método para compor automaticamente novas heurísticas a partir de um conjunto de heurísticas existentes; e um método para a construção de heurísticas baseadas em dados oriundos da análise de mutação de programas. Os aspectos inovadores de ambos os métodos referem-se à investigação conjunta de: (i) especialização de heurísticas para determinados programas; (ii) aplicação de abordagem evolutiva para a geração de heurísticas com equações não lineares; (iii) criação de heurísticas a partir da combinação de heurísticas tradicionais; (iv) uso de espectro de cobertura e de mutação extraídos da atividade de teste; (v) análise e comparação da eficácia de métodos que usam os espectros de cobertura e de mutação para a localização de defeitos; e (vi) análise da qualidade dos espectros de mutação como fonte de dados para a localização de defeitos. Os resultados apontaram competitividade de ambas as abordagens em seus contextos.