Virginia Tech
Statistical Methods for Performance Evaluation of Machine Learning and Artificial Intelligence Models
Abstract
dc:description.abstractArtificial Intelligence (AI) and Machine Learning (ML) have revolutionized the field of statistics by introducing advanced data analysis methods for predictive modeling and decision- making. These technologies facilitate handling both structured and unstructured data with high accuracy and efficiency. However, the performance evaluation of AI and ML models poses unique statistical challenges, especially in complex and unbalanced data scenarios. This dissertation aims to develop and refine statistical methodologies for assessing and improv- ing the efficacy of AI and ML models, with a focus on practical applications in solar panel electroluminescence (EL) image analysis under unbalanced dataset scenarios. Additionally, considering the growing popularity of large language models (LLMs) and their applications in AI-powered statistical analysis, metrics and performance evaluations on AI-generated code quality are explored for actionable insights. Chapter 1 begins with an introduction to EL image scenarios. It discusses the ML models used for identifying defects in EL images and explains the metrics utilized for performance evaluation. This chapter also introduces the analysis under unbalanced data scenarios, ex- plores AI-powered statistical programming and methods of quality control. Chapter 2 provides an in-depth analysis of various machine learning and deep learning (DL) methods applied to the classification of solar panel EL images. We compare these methods using different performance metrics to assess their effectiveness in classifying EL images of solar cells. To evaluate the robustness of ML models, we perform replications of the experiments. Additionally, we provide a thorough discussion on the selection and impact of various assessment criteria. The results offer insights and guidelines for practitioners in choosing appropriate prediction methods and evaluation metrics. Chapter 3 focuses on improving the accuracy and robustness of machine learning models when dealing with unbalanced data scenarios. Building on the comparison study from Chap- ter 2, we found a severe imbalance in EL image datasets: functional and severely defective classes are the majority, while mildly and moderately defective classes are the minority. This imbalance harms model performance in rare event prediction and robustness. To address this issue, sampling techniques and multiple customized loss functions are employed by choosing dynamicweightsatclassandsampleleveltoenhancetheaccuracyandreliabilityofpredictions for the minority classes. Chapter4introducestheincreasingpopularityandcriticalroleofLLMs, suchasChatGPT and Llama, in automating statistical programming tasks within SAS programming. Despite the strengths of LLMs in generating syntactically correct code, there still remain concerns regarding their ability to handle complex statistical analyses that require deep domain-specific expertise and ensure executable code. To address these issues, a systematic evaluation of SAS code generated by various LLMs through multiple dimensions of assessments is conducted by human experts. This evaluation specifically examines correctness, effectiveness, readability, executability, and accuracy of code execution results. This investigation aims to highlight the capabilities and limitations of LLM-generated code, providing actionable insights into their effective integration into statistical programming workflows. Chapter 5 introduces StatLLM, an open-source benchmark dataset designed to evaluate the coding capabilities of LLMs in statistical analysis tasks. Recognizing the lack of stan- dardized datasets for assessing statistical code, StatLLM addresses this gap by providing comprehensive statistical analysis tasks, corresponding SAS code generated by popular LLMs (ChatGPT 3.5, ChatGPT 4.0, and Llama 3.1), and detailed human expert evaluations of code correctness, effectiveness, readability, executability, and output accuracy. Derived from the evaluations presented in Chapter 4, StatLLM offers a valuable resource for advancing natural language processing metrics, improving LLM performance, and guiding the development of next-generation statistical software, thus enhances critical advancements in machine learning and data science research. Chapter 6 provides the summary of key results of this dissertation and explores the direc- tions for future research opportunities.
Degree
thesis:*- Name thesis:degree_name
- Doctor of Philosophy
- Level thesis:degree_level
- doctoral
- Discipline thesis:degree_discipline
- Statistics
- Department dc:contributor.department
- Statistics
- Grantor dc:publisher
- Virginia Tech
- Year dc:date.issued
- 2025
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Song, Xinyi
- Chair dc:contributor.committeechair
-
- Hong, Yili
- Committee members dc:contributor.committeemember
-
- Freeman, Laura J.
- Deng, Xinwei
- Xing, Xin
Subjects
dc:subject × 6Rights
dc:rights- Statement dc:rights
-
- In Copyright
- Licence dc:rights.uri
- Language dc:language.iso
- en
Identifiers
dc:identifier.*- Dc Identifier Other
- vt_gsexam:44183