Georgia Institute of Technology
Risk-Aware Data-Driven Simulation: Uncertainty Quantification, Model Validation, and Optimization
Abstract
dc:description.abstractSimulation serves as a powerful methodology for analyzing complex stochastic systems, especially when analytical solutions are intractable. A fundamental but often implicit assumption underlying simulation-based analysis is that the simulation model accurately characterizes the behavior of the real-world system it represents. However, in practice, this assumption is rarely satisfied and some components (e.g., parameters) of the simulator are unknown and thus, must be estimated from data. In this dissertation, we adopt "data-driven simulation" as an umbrella term to refer to a class of stochastic simulation models that are calibrated/validated from real-world data. Due to finiteness of the data, the analysis based on data-driven simulation is subject to model risk stemming from discrepancy between the simulation model and the target system. This dissertation explores methodological frameworks for simulation-based inference and optimization in the presence of model risk, with a focus on three core challenges: uncertainty quantification, simulation model validation, and optimization under uncertainty. Chapter 2 addresses the uncertainty quantification of a simulation model where its stochastic input-generating processes are estimated from data. Especially, we focus on performance measures that can be expressed as a ratio of two dependent simulation outputs' means. We employ the parametric bootstrap approach to construct percentile confidence intervals (CIs) for the ratio under the correct input models. A standard ratio estimator that takes the ratio of two sample means may have a large bias and variance when the simulation budget is small, leading to a CI with a poor coverage probability. We propose two new estimators: the kNN estimator pools information via k-nearest neighbor regression and performs well in low dimensions, and the kLR estimator extends the kNN approach by combining the likelihood ratio (LR) method to achieve better convergence rates and finite variance even in high-dimensional settings. We analyze the asymptotic efficiency of the estimators and propose finite-sample heuristics to improve the experiment design. Empirical studies demonstrate their superior performances compared to the standard approach. Chapter 3 studies simulation model validation when the simulator's and system's key performance indicators (KPIs) are high-dimensional and evolve with time. We assume that the system KPIs are observed at the end of several finite-length epochs alongside with the system states at the beginning of the epochs. The simulator takes the observed system state and initializes its own state to generate the KPIs at the end of each epoch. Unlike the system KPIs, which are observed only once in each epoch, the simulator can generate multiple sample paths of the simulated KPIs. Merging these two data sets, we propose a two-part hypothesis testing procedure: marginal tests for checking the distributional alignment of each KPI across all epochs, and a joint test for assessing whether the dependence structure, modeled by copulas, matches. The approach uses probability integral transform and nonparametric copula estimation, and establishes asymptotic guarantees on Type-I error. To enhance finite-sample reliability, a stepdown multiple testing procedure with bootstrap-derived critical values is introduced. Chapter 4 considers a simulation optimization problem under input uncertainty, where the parameters of the simulator's input models are updated using streaming data obtained in batches over multiple periods from the system. Since decisions must be made in each period, the chapter formulates a multi-period stochastic approximation (SA) framework that iteratively refines decisions based on increasingly accurate parameter estimates. Assuming strong convexity of the objective function, two algorithms are proposed. Re-start SA (ReSA) resets its stepsizes each period and takes just enough number of steps so that the SA error matches the input uncertainty error. Warm-start SA (WaSA) improves the computational efficiency of ReSA by adaptively reducing the stepsizes for later periods. Both are shown to achieve optimal convergence under some regularity conditions. A regularized version of ReSA is also introduced to relax an assumption on the known strong convexity parameter.
Degree
thesis:*- Level thesis:degree_level
- Doctoral
- Department dc:contributor.department
- Industrial and Systems Engineering
- Grantor dc:publisher
- Georgia Institute of Technology
- Year dc:date.issued
- 2025
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- He, Linyun
- Advisor dc:contributor.advisor
-
- Song, Eunhye
- Committee members dc:contributor.committeemember
-
- Joseph, Roshan
- Kim, Seong-Hee
- Zhou, Enlu
- Shanbhag, Uday
Subjects
dc:subject × 3Rights
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/82079