<p dir="ltr">Approaches to system testing and evaluation (T&E) are becoming increasingly relevant as artificial intelligence (AI)/machine learning (ML) technology expands across the industry’s current landscape. As the AI/ML landscape continues to develop, greater amounts of data are required to build the next generation of technology. Multiple communities have worked to create frameworks to interact with such scales of data, yet a gap persists in the ability to utilize data generated throughout the development process to support the for use in a T&E program. The objective of this thesis is to address this gap through a multi-fidelity approach to the test and evaluation of AI-enabled systems. This approach is constructed using a space of models to visualize similarities and differences between each individual model. Once requirements and potential tests that models can be employed to fulfill are organized, a method to sequentially select models for testing is utilized. Models are selected to maximize utility, dependent on model performance and cost to the T&E team. Experimentation was conducted through the case of an autonomous vehicle (AV) perception system, where models were constructed using a simulation of the Purdue University campus for AVs to drive around. Results show that the proposed approach, when paired with Bayesian Optimization for sequential test selection through an expected improvement acquisition function, can effectively select models in a manner that works to minimize uncertainty and cost for the test team. Through computational experiments, the proposed approach can be used to develop test combinations that minimize costs and maximize utility while maximizing the information a T&E team has on how well a system can meet a set of testing requirements in operational conditions.</p>
| Identifer | oai:union.ndltd.org:purdue.edu/oai:figshare.com:article/26364277 |
| Date | 27 July 2024 |
| Creators | Robert Joseph Seif (19206790) |
| Source Sets | Purdue University |
| Detected Language | English |
| Type | Text, Thesis |
| Rights | CC BY 4.0 |
| Relation | https://figshare.com/articles/thesis/A_Multi-Fidelity_Approach_to_Testing_and_Evaluation_of_AI-Enabled_Systems/26364277 |
Page generated in 0.0025 seconds