Back to search

Cornell University

Reconsidering Ranking-and-Selection Guarantees

Abstract

dc:description.abstract

This dissertation deals with the various statistical guarantees delivered by ranking-and-selection (R&S) procedures: a class of methods designed for the problem of selecting the best from among a finite number of simulated systems. Examples of such guarantees include ensuring that an optimal or near-optimal system is selected with high probability or that the expected performance gap between the selected system and the optimal system is below a specified threshold. We explore three fundamental issues concerning R&S guarantees that are of practical and theoretical interest to the simulation community. First, we discuss the shortcomings of the popular indifference-zone-inspired guarantee on the probability of correct selection (PCS) and argue that delivering a guarantee on the probability of good selection (PGS) is a more justifiable goal. We present an overview of the PGS guarantee and examine numerous techniques for proving the PGS guarantee, including sufficient conditions under which R&S procedures that deliver the IZ-inspired PCS guarantee also deliver the PGS guarantee. Second, we study Bayesian R&S guarantees, contrasting them with their frequentist counterparts and investigating the practical implications of this distinction. R&S procedures deliver Bayesian guarantees by terminating when a posterior quantity of interest - e.g., the posterior PCS or PGS - crosses some threshold. We develop several methods for improving the computational efficiency of checking this stopping rule when there are a large number of systems and demonstrate their effectiveness compared to existing approaches. Third, we study R&S guarantees for the setting in which a R&S procedure is run after a simulation-optimization search. We show that for searches that use the observed performance of explored systems to identify new systems, the simulation replications are conditionally dependent given the sequence of returned systems. We demonstrate that reusing replications taken during a search as input to a R&S procedure can result in an empirical PCS or PGS below the guaranteed threshold. Based on these negative findings, we call into question the guarantees of established R&S procedures that reuse search data.

Degree

thesis:*
Name thesis:degree_name
Ph.D., Operations Research and Information Engineering
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Operations Research and Information Engineering
Grantor
Cornell University
Year dc:date.issued
2019

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Eckman, David
Chair dc:contributor.chair
  • Henderson, Shane G.
Committee members dc:contributor.committeemember
  • Frazier, Peter
  • Bitar, Eilyan Yamen

Subjects

dc:subject × 5

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Dc Identifier Bibid
11050367
Dc Identifier Other
ProQuest Submission ID: 11459
ProQuest Publication ID: 13883805

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Eckman, David. Reconsidering Ranking-and-Selection Guarantees. Doctor of Philosophy thesis, Cornell University, 2019. https://hdl.handle.net/1813/67385