{"id":{"repo_id":"denver","oai_identifier":"oai:digitalcommons.du.edu:etd-1599"},"canonical_url":"https://search.ndltd.org/etd/denver/oai:digitalcommons.du.edu:etd-1599","repository":{"repo_id":"denver","name":"University of Denver","base_url":"https://digitalcommons.du.edu/do/oai/"},"display":{"title":"An Empirical Evaluation of the Effectiveness of JML Assertions as Test Oracles","abstract":"Test oracles remain one of the least understood aspects of the modern testing process. An oracle is a mechanism used by software testers and software engineers for determining whether a test has passed or failed. One widely-supported approach to oracles is the use of runtime assertion checking during the testing activity. Method invariants,pre- and postconditions help detect bugs during runtime. While assertions are supported by virtually all programming environments, are used widely in practice, and are often assumed to be effective as test oracles, there are few empirical studies of their efficacy in this role. In this thesis, we present the results of an experiment we conducted to help understand this question. To do this, we studied seven of the core Java classes that had been annotated by others with assertions in the Java Modeling Language, used the muJava mutation analysis tool to create mutant implementations of these classes, exercised them with input-only (i.e., no oracle) test suites that achieve branch coverage, and used a machine learning tool, Weka, to determine which annotations were effective at ``killing'' these mutants. We also evaluate how effective the ``null oracle'' (in our case, the Java runtime system) is at catching these bugs. The results of our study are interesting, and help provide software engineers with insight into situations in which assertions can be relied upon to find bugs, and situations in which assertions may need to be augmented with other approaches to test oracles.","abstract_html":"Test oracles remain one of the least understood aspects of the modern testing process. An oracle is a mechanism used by software testers and software engineers for determining whether a test has passed or failed. One widely-supported approach to oracles is the use of runtime assertion checking during the testing activity. Method invariants,pre- and postconditions help detect bugs during runtime. While assertions are supported by virtually all programming environments, are used widely in practice, and are often assumed to be effective as test oracles, there are few empirical studies of their efficacy in this role. In this thesis, we present the results of an experiment we conducted to help understand this question. To do this, we studied seven of the core Java classes that had been annotated by others with assertions in the Java Modeling Language, used the muJava mutation analysis tool to create mutant implementations of these classes, exercised them with input-only (i.e., no oracle) test suites that achieve branch coverage, and used a machine learning tool, Weka, to determine which annotations were effective at ``killing&#x27;&#x27; these mutants. We also evaluate how effective the ``null oracle&#x27;&#x27; (in our case, the Java runtime system) is at catching these bugs. The results of our study are interesting, and help provide software engineers with insight into situations in which assertions can be relied upon to find bugs, and situations in which assertions may need to be augmented with other approaches to test oracles.","abstract_has_math":false,"creators":["Shrestha, Kavir"],"institution":null,"degree_name":"M.S.","degree_level":"Masters Thesis","degree_discipline":null,"degree_department":null,"school":null,"contributors":["Matthew J. Rutherford, Ph.D.","Susanne Sherba","Mohammad H. Mahoor"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":null,"date_issued":"","date_published":null,"updated_at":"2026-11-19T07:54:47Z","subjects":["Design by contract","Empirical study","Runtime assertion checking","Test oracles","Computer Engineering","Other Computer Engineering"],"languages":["en"],"rights":["<p>Copyright is held by the author. User is responsible for all copyright compliance.</p>","Kavir Shrestha"],"rights_urls":[],"identifier_entries":[],"person_identifier_entries":[]},"links":{"outbound_url":"https://digitalcommons.du.edu/etd/600","outbound_label":"Repository record","outbound_source":"dc:identifier","record_url":"https://digitalcommons.du.edu/etd/600","record_label":"Repository record","record_kind":"record","record_source":"dc:identifier","download_url":null,"download_label":null,"download_source":null,"access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://digitalcommons.du.edu/do/oai/?verb=GetRecord&metadataPrefix=oai_etdms&identifier=oai%3Adigitalcommons.du.edu%3Aetd-1599","prefix":"oai_etdms"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Matthew J. Rutherford, Ph.D.","Susanne Sherba","Mohammad H. Mahoor"]},{"key":"dc:creator","label":"Author","values":["Shrestha, Kavir"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.available","label":"Dc Date Available","values":["2001-01-01T08:00:00Z"]},{"key":"dc:date.copyright","label":"Dc Date Copyright","values":["2010-01-01T08:00:00Z"]},{"key":"dc:date.created","label":"Dc Date Created","values":["2010-01-01T08:00:00Z"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Masters Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Design by contract","Empirical study","Runtime assertion checking","Test oracles","Computer Engineering","Other Computer Engineering"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["<p>Copyright is held by the author. User is responsible for all copyright compliance.</p>"]},{"key":"dc:rights.holder","label":"Dc Rights Holder","values":["Kavir Shrestha"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://digitalcommons.du.edu/etd/600"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["Test oracles remain one of the least understood aspects of the modern testing process. An oracle is a mechanism used by software testers and software engineers for determining whether a test has passed or failed. One widely-supported approach to oracles is the use of runtime assertion checking during the testing activity. Method invariants,pre- and postconditions help detect bugs during runtime. While assertions are supported by virtually all programming environments, are used widely in practice, and are often assumed to be effective as test oracles, there are few empirical studies of their efficacy in this role. In this thesis, we present the results of an experiment we conducted to help understand this question. To do this, we studied seven of the core Java classes that had been annotated by others with assertions in the Java Modeling Language, used the muJava mutation analysis tool to create mutant implementations of these classes, exercised them with input-only (i.e., no oracle) test suites that achieve branch coverage, and used a machine learning tool, Weka, to determine which annotations were effective at ``killing'' these mutants. We also evaluate how effective the ``null oracle'' (in our case, the Java runtime system) is at catching these bugs. The results of our study are interesting, and help provide software engineers with insight into situations in which assertions can be relied upon to find bugs, and situations in which assertions may need to be augmented with other approaches to test oracles."]},{"key":"dc:description.provenance","label":"Dc Description Provenance","values":["Received from ProQuest"]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:format.extent","label":"Dc Format Extent","values":["76 p."]},{"key":"dc:title","label":"Title","values":["An Empirical Evaluation of the Effectiveness of JML Assertions as Test Oracles"]}]}],"canonical_facts":{"dc:contributor":["Matthew J. Rutherford, Ph.D.","Susanne Sherba","Mohammad H. Mahoor"],"dc:creator":["Shrestha, Kavir"],"dc:date.available":["2001-01-01T08:00:00Z"],"dc:date.copyright":["2010-01-01T08:00:00Z"],"dc:date.created":["2010-01-01T08:00:00Z"],"dc:description.abstract":["Test oracles remain one of the least understood aspects of the modern testing process. An oracle is a mechanism used by software testers and software engineers for determining whether a test has passed or failed. One widely-supported approach to oracles is the use of runtime assertion checking during the testing activity. Method invariants,pre- and postconditions help detect bugs during runtime. While assertions are supported by virtually all programming environments, are used widely in practice, and are often assumed to be effective as test oracles, there are few empirical studies of their efficacy in this role. In this thesis, we present the results of an experiment we conducted to help understand this question. To do this, we studied seven of the core Java classes that had been annotated by others with assertions in the Java Modeling Language, used the muJava mutation analysis tool to create mutant implementations of these classes, exercised them with input-only (i.e., no oracle) test suites that achieve branch coverage, and used a machine learning tool, Weka, to determine which annotations were effective at ``killing'' these mutants. We also evaluate how effective the ``null oracle'' (in our case, the Java runtime system) is at catching these bugs. The results of our study are interesting, and help provide software engineers with insight into situations in which assertions can be relied upon to find bugs, and situations in which assertions may need to be augmented with other approaches to test oracles."],"dc:description.provenance":["Received from ProQuest"],"dc:format":["application/pdf"],"dc:format.extent":["76 p."],"dc:identifier":["https://digitalcommons.du.edu/etd/600"],"dc:language":["en"],"dc:rights":["<p>Copyright is held by the author. User is responsible for all copyright compliance.</p>"],"dc:rights.holder":["Kavir Shrestha"],"dc:subject":["Design by contract","Empirical study","Runtime assertion checking","Test oracles","Computer Engineering","Other Computer Engineering"],"dc:title":["An Empirical Evaluation of the Effectiveness of JML Assertions as Test Oracles"],"thesis:degree_level":["Masters Thesis"],"thesis:degree_name":["M.S."]},"updated_at":"2026-11-19T07:54:47Z"}