Return to search

Homogeneity Test on Error Rates from Ordinal Scores and Application to Forensic Science

The Receiver Operating Characteristic (ROC) curve is used to measure the classification accuracy of tests that yield ordinal or continuous scores. Ordinal scores are common in medical imaging studies and, more recently, in black-box studies on forensic identification accuracy (Phillips et al., 2018). To assess the accuracy of radiologists in medical imaging studies or the accuracy of forensic examiners in biometric studies, one needs to estimate the ROC curves from the ordinal scores and account for the covariates related to the radiologists or forensic examiners. In this thesis, we propose a homogeneity test to compare the performance of raters. We derive the asymptotic properties of estimated ROC curves and their corresponding Area Under the Curve (AUC) within an ordinal regression framework. Moreover, we investigate differences in ROC curves (and AUCs) among examiners in detail. We construct confidence intervals for the difference in AUCs and confidence bands for the difference in ROC curves for performance comparison purposes. First, we conduct simulations on data where scores are assumed to be normally distributed, and the features include both categorical and continuous covariates. Then, we apply our procedure to facial recognition data to compare forensic examiners.
The second part of this thesis addresses the correlation of decision scores among raters. In medical imaging studies and facial recognition, multiple raters assess the same subject pairs, leading to potential score correlations. Because of these correlated scores, standard methods for generalized linear models cannot be directly applied to estimate accuracy. In this thesis, we employ the generalized estimating equation to estimate covariate-specific and covariate-adjusted AUC values when correlations are present in ordinal scores. We conduct homogeneity tests on both covariate-specific and covariate-adjusted AUCs, investigating their statistical properties. To assess the finite sample properties of the test, we conduct simulation studies. Furthermore, we apply this test to real facial recognition data.

Identiferoai:union.ndltd.org:ucf.edu/oai:stars.library.ucf.edu:etd2023-1015
Date01 January 2023
CreatorsNguyen, Ngoc Ty
PublisherSTARS
Source SetsUniversity of Central Florida
LanguageEnglish
Detected LanguageEnglish
Typetext
Formatapplication/pdf
SourceGraduate Thesis and Dissertation 2023-2024

Page generated in 0.0021 seconds