University of Sheffield
Investigating the Relationships Between Mutants, Oracles and Pseudo-Testedness
Abstract
dc:description.abstractTest suites are critical in developing high-quality software, yet are commonly measured using code coverage. Code coverage measures the quantity of test suite-executed code, without requiring code behaviours be checked by an oracle (e.g., an assertion), leaving code vulnerable to pseudo-testedness. Pseudo-tested code is executed by the test suite, yet can be deleted without triggering test failures. Intuitively, pseudo-testedness can be detected using deletion operators, but has not been explored at the statement level. The relationship between pseudo-tested statement identification (PTSI) and other oracle- based adequacy metrics, i.e., checked coverage, remains unclear, leaving open questions about each technique’s role. Because PTSI relies on rule-based deletion mutation, it only highlights the absence of a check. To move beyond identifying where checks are missing, mutants must produce semantically incorrect behaviours that reveal the context requiring the check. While large language models (LLMs) enable context-aware mutation testing, current LLM-based tools are constrained to small code changes, such as lines or AST nodes. To utilise the LLM’s capabilities, tools must move towards algorithm- level approaches that guide the LLM to produce non-trivial, productive mutants. This thesis aims to characterise these relationships and utilise them to refine the techniques. First, this thesis explores pseudo-testedness across 23 projects, identifying that pseudo- tested methods miss 48% of pseudo-tested statements. Secondly, compared with checked coverage using dynamic and observation-based slicers, PTSI locates code with the lowest median mutation scores (0.32 vs. 0.76 and 0.5) while executing at least 10x faster. Finally, this thesis uses systematic hazard analysis to guide LLMs in generating algorithm-level mutants that are uniquely subsuming and defect-matching compared with existing LLM- and rule-based techniques. By characterising the relationships between mutants, oracles and pseudo-testedness, this thesis enables the refinement of these techniques for test suite assessment.
Degree
thesis:*- Name dc:type.qualificationname
- Ph.D
- Level dc:type.qualificationlevel
- doctoral
- Grantor dc:publisher.institution
- University of Sheffield
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Maton, Megan
- Advisor dc:contributor.advisor
-
- McMinn, Phil
- ORCID person:identifier.orcid
Rights
- Language dc:language.dcterms_iso639_2
- eng
Identifiers
dc:identifier.*- Dc Identifier Grantnumber
- EP/W524360/1