Abstract
dc:description.abstractAutomated software testing is a popular method of quality control that aims to detect bugs before software is released to the end user. Unfortunately, writing, maintaining, and executing automated test suites is expensive and not necessarily cost effective. To gain a better understanding of the return on investment and cost of automated testing, we studied the continuous integration build results of 61 open source projects. We found that 19.4% of build failures are resolved by a change to only test code. These failures do not find bugs in the code under test, but rather bugs in the test suites themselves and represent a maintenance cost. We also found that 12.8% of the build failures we studied are due to non-deterministic tests that can be difficult to debug. To gain a better understanding of what leads to test maintenance we manually inspected both valuable and costly tests and found a variety of reasons for test maintenance. Our empirical analysis of real projects quantifies the cost of maintaining test suites and shows that reducing maintenance costs could make automated testing much more affordable.
Degree
thesis:*- Grantor dc:publisher
- University of Waterloo
- Year dc:date.issued
- 2016
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Labuschagne, Adriaan
- Advisors dc:contributor.advisor
-
- Holmes, Reid
- Godfrey, Michael
Subjects
dc:subject × 1Rights
- Language dc:language.iso
- en
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- http://hdl.handle.net/10012/10683