{"id":{"repo_id":"marshall","oai_identifier":"oai:mds.marshall.edu:etd-2827"},"canonical_url":"https://search.ndltd.org/etd/marshall/oai:mds.marshall.edu:etd-2827","repository":{"repo_id":"marshall","name":"Marshall University","base_url":"https://mds.marshall.edu/do/oai/"},"display":{"title":"Leveraging Explainable Artificial Intelligence (XAI) to Understand Performance Deviations in Load Tests of Large Software Systems","abstract":"Performance testing generates vast amounts of data, making it challenging for human analysts to process within a reasonable timeframe. Therefore, black-box machine learning models are often used to determine pass/fail status, but these models lack transparency and cannot explain why a test has failed, leading to a time-consuming manual analysis process. To address this issue, this thesis proposes using Explainable Artificial Intelligence (XAI) to improve the trustworthiness of black-box and interpretable models in performance testing. The proposed approach leverages the Shapley Additive Explanation (SHAP) algorithm as a surrogate model to help performance analysts understand the decision-making process of black-box machine learning models. By wrapping SHAP around black-box models, analysts can gain explainability on why a model predicted a test's pass or fail status and identify the relative importance of performance data to machine learning models. The proposed approach was evaluated through several load text experiments on a real testbed, using industry-standard performance benchmarks, manually injecting performance bugs into the system to synthesize ground truth and building machine learning models using a black box learner (Artificial Neural Network) and an interpretable learner (Random Forest) to predict the test's pass or fail status. The results demonstrate that classical performance measures such as precision, recall, F-measure, and accuracy are not sufficient to gauge the reliability and trustworthiness of machine learning models. Instead, the proposed approach stands out by providing the explanations behind the decisions made by learning algorithms and enhancing their trustworthiness. The proposed approach, though evaluated through load testing can be generalized to other domains and require little or no effort to operate.","abstract_html":"Performance testing generates vast amounts of data, making it challenging for human analysts to process within a reasonable timeframe. Therefore, black-box machine learning models are often used to determine pass/fail status, but these models lack transparency and cannot explain why a test has failed, leading to a time-consuming manual analysis process. To address this issue, this thesis proposes using Explainable Artificial Intelligence (XAI) to improve the trustworthiness of black-box and interpretable models in performance testing. The proposed approach leverages the Shapley Additive Explanation (SHAP) algorithm as a surrogate model to help performance analysts understand the decision-making process of black-box machine learning models. By wrapping SHAP around black-box models, analysts can gain explainability on why a model predicted a test&#x27;s pass or fail status and identify the relative importance of performance data to machine learning models. The proposed approach was evaluated through several load text experiments on a real testbed, using industry-standard performance benchmarks, manually injecting performance bugs into the system to synthesize ground truth and building machine learning models using a black box learner (Artificial Neural Network) and an interpretable learner (Random Forest) to predict the test&#x27;s pass or fail status. The results demonstrate that classical performance measures such as precision, recall, F-measure, and accuracy are not sufficient to gauge the reliability and trustworthiness of machine learning models. Instead, the proposed approach stands out by providing the explanations behind the decisions made by learning algorithms and enhancing their trustworthiness. The proposed approach, though evaluated through load testing can be generalized to other domains and require little or no effort to operate.","abstract_has_math":false,"creators":["Shoemaker, Eric"],"institution":null,"degree_name":"Computer Science","degree_level":"Thesis","degree_discipline":"College of Engineering and Computer Sciences","degree_department":null,"school":null,"contributors":["Dr. Jamil Chaudri, Committee Chairperson","Dr. Husnu Narman","Dr. Haroon Malik"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":null,"date_issued":"","date_published":null,"updated_at":"2026-11-21T19:58:01Z","subjects":["Artificial Intelligence and Robotics","Computer Sciences","Physical Sciences and Mathematics"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[],"person_identifier_entries":[]},"links":{"outbound_url":"https://mds.marshall.edu/etd/1822","outbound_label":"Repository record","outbound_source":"dc:identifier","record_url":"https://mds.marshall.edu/etd/1822","record_label":"Repository record","record_kind":"record","record_source":"dc:identifier","download_url":null,"download_label":null,"download_source":null,"access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://mds.marshall.edu/do/oai/?verb=GetRecord&metadataPrefix=oai_etdms&identifier=oai%3Amds.marshall.edu%3Aetd-2827","prefix":"oai_etdms"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Dr. Jamil Chaudri, Committee Chairperson","Dr. Husnu Narman","Dr. Haroon Malik"]},{"key":"dc:creator","label":"Author","values":["Shoemaker, Eric"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.available","label":"Dc Date Available","values":["2023-08-30T07:00:00Z"]},{"key":"dc:date.created","label":"Dc Date Created","values":["2023-01-01T08:00:00Z"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["College of Engineering and Computer Sciences"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Computer Science"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Artificial Intelligence and Robotics","Computer Sciences","Physical Sciences and Mathematics"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://mds.marshall.edu/etd/1822"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["Performance testing generates vast amounts of data, making it challenging for human analysts to process within a reasonable timeframe. Therefore, black-box machine learning models are often used to determine pass/fail status, but these models lack transparency and cannot explain why a test has failed, leading to a time-consuming manual analysis process. To address this issue, this thesis proposes using Explainable Artificial Intelligence (XAI) to improve the trustworthiness of black-box and interpretable models in performance testing. The proposed approach leverages the Shapley Additive Explanation (SHAP) algorithm as a surrogate model to help performance analysts understand the decision-making process of black-box machine learning models. By wrapping SHAP around black-box models, analysts can gain explainability on why a model predicted a test's pass or fail status and identify the relative importance of performance data to machine learning models. The proposed approach was evaluated through several load text experiments on a real testbed, using industry-standard performance benchmarks, manually injecting performance bugs into the system to synthesize ground truth and building machine learning models using a black box learner (Artificial Neural Network) and an interpretable learner (Random Forest) to predict the test's pass or fail status. The results demonstrate that classical performance measures such as precision, recall, F-measure, and accuracy are not sufficient to gauge the reliability and trustworthiness of machine learning models. Instead, the proposed approach stands out by providing the explanations behind the decisions made by learning algorithms and enhancing their trustworthiness. The proposed approach, though evaluated through load testing can be generalized to other domains and require little or no effort to operate."]},{"key":"dc:title","label":"Title","values":["Leveraging Explainable Artificial Intelligence (XAI) to Understand Performance Deviations in Load Tests of Large Software Systems"]}]}],"canonical_facts":{"dc:contributor":["Dr. Jamil Chaudri, Committee Chairperson","Dr. Husnu Narman","Dr. Haroon Malik"],"dc:creator":["Shoemaker, Eric"],"dc:date.available":["2023-08-30T07:00:00Z"],"dc:date.created":["2023-01-01T08:00:00Z"],"dc:description.abstract":["Performance testing generates vast amounts of data, making it challenging for human analysts to process within a reasonable timeframe. Therefore, black-box machine learning models are often used to determine pass/fail status, but these models lack transparency and cannot explain why a test has failed, leading to a time-consuming manual analysis process. To address this issue, this thesis proposes using Explainable Artificial Intelligence (XAI) to improve the trustworthiness of black-box and interpretable models in performance testing. The proposed approach leverages the Shapley Additive Explanation (SHAP) algorithm as a surrogate model to help performance analysts understand the decision-making process of black-box machine learning models. By wrapping SHAP around black-box models, analysts can gain explainability on why a model predicted a test's pass or fail status and identify the relative importance of performance data to machine learning models. The proposed approach was evaluated through several load text experiments on a real testbed, using industry-standard performance benchmarks, manually injecting performance bugs into the system to synthesize ground truth and building machine learning models using a black box learner (Artificial Neural Network) and an interpretable learner (Random Forest) to predict the test's pass or fail status. The results demonstrate that classical performance measures such as precision, recall, F-measure, and accuracy are not sufficient to gauge the reliability and trustworthiness of machine learning models. Instead, the proposed approach stands out by providing the explanations behind the decisions made by learning algorithms and enhancing their trustworthiness. The proposed approach, though evaluated through load testing can be generalized to other domains and require little or no effort to operate."],"dc:identifier":["https://mds.marshall.edu/etd/1822"],"dc:subject":["Artificial Intelligence and Robotics","Computer Sciences","Physical Sciences and Mathematics"],"dc:title":["Leveraging Explainable Artificial Intelligence (XAI) to Understand Performance Deviations in Load Tests of Large Software Systems"],"thesis:degree_discipline":["College of Engineering and Computer Sciences"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["Computer Science"]},"updated_at":"2026-11-21T19:58:01Z"}