When studying detection systems, parameters associated with the Receiver Operating Characteristic (ROC) curve are often estimated to assess system performance. In some applied settings it is often not possible to test the detection system with large numbers of stimuli. The resulting small sample statistics many have undesirable properties. The characteristics of these small sample ROC estimators were examined in a Monte Carlo simulation. Three popular ROC parameters were chosen for study. One of the parameters was a single parameter index of system performance, Area under the ROC curve. The other parameters, ROC intercept and slope, were considered as a pair. ROC intercept and slope were varied along with sample size and points on the certainty rating scale to form a four way factorial design. Several types of estimators were examined. For the parameter, Area under the curve, Maximum Likelihood (ML), three types of Least Squares (LS), and Distribution Free (DF) estimators were considered. Except for the DF estimator, the same estimators were considered for the parameters, intercept and slope. These estimators were compared with respect to three characteristics: bias, efficiency, and consistency. For Area under the curve, the ML estimator was the least biased. The DF estimator was the most efficient, and all the estimators except the DF estimator appeared to be consistent. For intercept and slope the LS estimator that minimized vertical error of the points from the ROC curve (line) was the least biased for both estimators. This LS estimator was also the most efficient. This estimator along with the ML estimator also appeared to be the most consistent. The other two estimators had no significant trend toward consistency. These results along with other findings, illustrate that different estimators may be "best" for different sample sizes and for different parameters. Therefore, researchers should carefully consider the characteristics of ROC estimators before using them as indices of system performance.
| Identifer | oai:union.ndltd.org:arizona.edu/oai:arizona.openrepository.com:10150/184127 |
| Date | January 1987 |
| Creators | BORGSTROM, MARK CRAIG. |
| Publisher | The University of Arizona. |
| Source Sets | University of Arizona |
| Language | English |
| Detected Language | English |
| Type | text, Dissertation-Reproduction (electronic) |
| Rights | Copyright © is held by the author. Digital access to this material is made possible by the University Libraries, University of Arizona. Further transmission, reproduction or presentation (such as public display or performance) of protected items is prohibited except with permission of the author. |
Page generated in 0.0023 seconds