Marshall University
Leveraging Explainable Artificial Intelligence (XAI) to Understand Performance Deviations in Load Tests of Large Software Systems
Abstract
dc:description.abstractPerformance testing generates vast amounts of data, making it challenging for human analysts to process within a reasonable timeframe. Therefore, black-box machine learning models are often used to determine pass/fail status, but these models lack transparency and cannot explain why a test has failed, leading to a time-consuming manual analysis process. To address this issue, this thesis proposes using Explainable Artificial Intelligence (XAI) to improve the trustworthiness of black-box and interpretable models in performance testing. The proposed approach leverages the Shapley Additive Explanation (SHAP) algorithm as a surrogate model to help performance analysts understand the decision-making process of black-box machine learning models. By wrapping SHAP around black-box models, analysts can gain explainability on why a model predicted a test's pass or fail status and identify the relative importance of performance data to machine learning models. The proposed approach was evaluated through several load text experiments on a real testbed, using industry-standard performance benchmarks, manually injecting performance bugs into the system to synthesize ground truth and building machine learning models using a black box learner (Artificial Neural Network) and an interpretable learner (Random Forest) to predict the test's pass or fail status. The results demonstrate that classical performance measures such as precision, recall, F-measure, and accuracy are not sufficient to gauge the reliability and trustworthiness of machine learning models. Instead, the proposed approach stands out by providing the explanations behind the decisions made by learning algorithms and enhancing their trustworthiness. The proposed approach, though evaluated through load testing can be generalized to other domains and require little or no effort to operate.
Degree
thesis:*- Name thesis:degree_name
- Computer Science
- Level thesis:degree_level
- Thesis
- Discipline thesis:degree_discipline
- College of Engineering and Computer Sciences
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Shoemaker, Eric
- Contributors dc:contributor
-
- Dr. Jamil Chaudri, Committee Chairperson
- Dr. Husnu Narman
- Dr. Haroon Malik
Subjects
dc:subject × 3Identifiers
dc:identifier.*- Repository record dc:identifier
- https://mds.marshall.edu/etd/1822