Back to search

University of Sheffield

Investigating the Relationships Between Mutants, Oracles and Pseudo-Testedness

Abstract

dc:description.abstract

Test suites are critical in developing high-quality software, yet are commonly measured using code coverage. Code coverage measures the quantity of test suite-executed code, without requiring code behaviours be checked by an oracle (e.g., an assertion), leaving code vulnerable to pseudo-testedness. Pseudo-tested code is executed by the test suite, yet can be deleted without triggering test failures. Intuitively, pseudo-testedness can be detected using deletion operators, but has not been explored at the statement level. The relationship between pseudo-tested statement identification (PTSI) and other oracle- based adequacy metrics, i.e., checked coverage, remains unclear, leaving open questions about each technique’s role. Because PTSI relies on rule-based deletion mutation, it only highlights the absence of a check. To move beyond identifying where checks are missing, mutants must produce semantically incorrect behaviours that reveal the context requiring the check. While large language models (LLMs) enable context-aware mutation testing, current LLM-based tools are constrained to small code changes, such as lines or AST nodes. To utilise the LLM’s capabilities, tools must move towards algorithm- level approaches that guide the LLM to produce non-trivial, productive mutants. This thesis aims to characterise these relationships and utilise them to refine the techniques. First, this thesis explores pseudo-testedness across 23 projects, identifying that pseudo- tested methods miss 48% of pseudo-tested statements. Secondly, compared with checked coverage using dynamic and observation-based slicers, PTSI locates code with the lowest median mutation scores (0.32 vs. 0.76 and 0.5) while executing at least 10x faster. Finally, this thesis uses systematic hazard analysis to guide LLMs in generating algorithm-level mutants that are uniquely subsuming and defect-matching compared with existing LLM- and rule-based techniques. By characterising the relationships between mutants, oracles and pseudo-testedness, this thesis enables the refinement of these techniques for test suite assessment.

Degree

thesis:*
Name dc:type.qualificationname
Ph.D
Level dc:type.qualificationlevel
doctoral
Grantor dc:publisher.institution
University of Sheffield
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Maton, Megan
Advisor dc:contributor.advisor
  • McMinn, Phil
ORCID person:identifier.orcid

Rights

Language dc:language.dcterms_iso639_2
eng

Identifiers

dc:identifier.*
Dc Identifier Grantnumber
EP/W524360/1

Chain of custody

source
Harvested from
White Rose University Consortium
Base URL
etheses.whiterose.ac.uk/cgi/oai2
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
related terms
citation

Maton, Megan. Investigating the Relationships Between Mutants, Oracles and Pseudo-Testedness. doctoral thesis, University of Sheffield, 2026.