Back to search

University of British Columbia

A study of the influence of assertions and mutants on test suite effectiveness

Abstract

dc:description

Test suite effectiveness is measured by assessing the portion of faults that can be detected by tests. To precisely measure a test suite’s effectiveness, one need to pay attention to both tests and the set of faults used. Code coverage is a popular test adequacy criterion in practice. Code coverage, however, remains controversial as there is a lack of coherent empirical evidence for its relation with test suite effectiveness. More recently, test suite size has been shown to be highly correlated with effectiveness. However, previous studies treat test methods as the smallest unit of interest, and ignore potential factors influencing the correlation between test suite size and test suite effectiveness. We propose to go beyond test suite size, by investigating test assertions inside test methods. First, we empirically evaluate the relationship between a test suite’s effectiveness and the (1) number of assertions, (2) assertion coverage, and (3) different types of assertions. We compose 6,700 test suites in total, using 24,000 assertions of five real-world Java projects. We find that the number of assertions in a test suite strongly correlates with its effectiveness, and this factor positively influences the relationship between test suite size and effectiveness. Our results also indicate that assertion coverage is strongly correlated with effectiveness. Second, instead of only focusing on the testing side, we propose to investigate test suite effectiveness also by considering fault types (the ways faults are generated) and faults in different types of statements. Measuring a test suite’s effectiveness can be influenced by using faults with different characteristics. Assessing test suite effectiveness without paying attention to the distribution of faults is not precise. Our results indicate that fault type and statement type where the fault is located can significantly influence a test suite’s effectiveness.

Degree

thesis:*
Name thesis:degree_name
Master of Applied Science - MASc
Level thesis:degree_level
master's
Discipline thesis:degree_discipline
Electrical and Computer Engineering
Grantor dc:publisher
University of British Columbia
Year dc:date
2016

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Zhang, Yucheng

Rights

dc:rights
Statement dc:rights
  • Attribution-NonCommercial-NoDerivatives 4.0 International
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/59417

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Zhang, Yucheng. A study of the influence of assertions and mutants on test suite effectiveness. master's thesis, University of British Columbia, 2016. http://hdl.handle.net/2429/59417