Back to search

University of British Columbia

Evaluating the quality of student-written software tests with curated mutation analysis

Abstract

dc:description

An important learning outcome in software engineering education is the ability to write an effective test suite that rigorously tests a target application. The standard approach for assessing test suites is to check coverage which can be problematic because coverage rewards code invocation, without checking test assertion correctness. Mutation Analysis (injecting a small fault into a clone of a codebase) has been used in both industry and academia to check test suite quality. A mutant is killed if any tests in the test suite fail on the clone. More mutants killed indicates a stronger suite, as it is more sensitive to defects. Mutation Analysis has been limited in an educational setting because of the prohibitive cost in both time and compute power to run the students' suites over all generated clones. We employed Mutation Analysis to assess test suite quality in our upper-year Software Engineering course at a large research intensive university. This paper makes two contributions: (1) We show that it is feasible and effective to use a small sample of hand-written mutants for grading, and (2) We assess effectiveness for promoting student learning by comparing students graded with coverage to those graded with Mutation Analysis. We found that mutation graded students write more correct tests, check more of the behaviour of invoked code, and more actively seek to understand the project specification.

Degree

thesis:*
Name thesis:degree_name
Master of Science - MSc
Level thesis:degree_level
master's
Discipline thesis:degree_discipline
Computer Science
Grantor dc:publisher
University of British Columbia
Year dc:date
2022

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Hall, Braxton
Contributors dc:contributor
  • Baniassad, Elisa
  • Holmes, Reid

Rights

dc:rights
Statement dc:rights
  • Attribution-NonCommercial-NoDerivatives 4.0 International
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/83142

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Hall, Braxton. Evaluating the quality of student-written software tests with curated mutation analysis. master's thesis, University of British Columbia, 2022. http://hdl.handle.net/2429/83142